==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 4 of 8):
2023-08-01 05:53:02.385596: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Running tests under Python 3.11.1: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3
[ RUN      ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker
INFO:tensorflow:Using local port 45583
I0801 05:53:10.835376 281472910916288 test_util.py:3813] Using local port 45583
INFO:tensorflow:Using local port 39015
I0801 05:53:10.836475 281472910916288 test_util.py:3813] Using local port 39015
INFO:tensorflow:Using local port 34775
I0801 05:53:10.836863 281472910916288 test_util.py:3813] Using local port 34775
INFO:tensorflow:Using local port 43165
I0801 05:53:10.837246 281472910916288 test_util.py:3813] Using local port 43165
2023-08-01 05:53:12.563346: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-01 05:53:12.709010: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
INFO:tensorflow:Cluster starting.
I0801 05:53:15.457952 281472910916288 failure_handler_test.py:432] Cluster starting.
[worker-0]:    I0801 05:53:15.951232 281473433893568 multi_process_runner.py:840] Subprocess with PID 2912431 (worker, 0) is now being started.
[worker-2]:    I0801 05:53:15.987754 281473433893568 multi_process_runner.py:840] Subprocess with PID 2912443 (worker, 2) is now being started.
[worker-1]:    I0801 05:53:16.013157 281473433893568 multi_process_runner.py:840] Subprocess with PID 2912437 (worker, 1) is now being started.
[worker-0]:    I0801 05:53:15.951648 281473433893568 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45583", "localhost:39015", "localhost:34775", "localhost:43165"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-2]:    I0801 05:53:15.988165 281473433893568 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45583", "localhost:39015", "localhost:34775", "localhost:43165"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:16.038178 281473433893568 multi_process_runner.py:840] Subprocess with PID 2912452 (worker, 3) is now being started.
[worker-3]:    I0801 05:53:16.038599 281473433893568 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45583", "localhost:39015", "localhost:34775", "localhost:43165"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-1]:    I0801 05:53:16.013569 281473433893568 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45583", "localhost:39015", "localhost:34775", "localhost:43165"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-0]:    2023-08-01 05:53:16.068969: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:45583
[worker-0]:    2023-08-01 05:53:16.126286: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 14967891362734456343
[worker-0]:    2023-08-01 05:53:16.126749: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    2023-08-01 05:53:16.169476: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34775
[worker-1]:    2023-08-01 05:53:16.176878: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39015
[worker-0]:    2023-08-01 05:53:16.227275: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 3016094793347499796
[worker-2]:    2023-08-01 05:53:16.249066: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:16.257850: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 15392830185674010230
[worker-1]:    2023-08-01 05:53:16.265590: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    2023-08-01 05:53:16.310142: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:43165
[worker-0]:    2023-08-01 05:53:16.314037: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 2341262884183615680
[worker-3]:    2023-08-01 05:53:16.314256: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    I0801 05:53:16.315957 281473433893568 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-3]:    I0801 05:53:16.316075 281473433893568 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    I0801 05:53:16.316025 281473433893568 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:16.315958 281473433893568 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    I0801 05:53:16.382963 281473433893568 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:16.383500 281473433893568 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:16.383733 281473433893568 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-0]:    I0801 05:53:16.400079 281473433893568 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    I0801 05:53:16.400637 281473433893568 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:16.400871 281473433893568 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    I0801 05:53:16.416887 281473433893568 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:16.417415 281473433893568 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:16.417647 281473433893568 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    I0801 05:53:16.474779 281473433893568 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    I0801 05:53:16.475362 281473433893568 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:16.475594 281473433893568 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45583', 'localhost:39015', 'localhost:34775', 'localhost:43165']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-2]:    I0801 05:53:16.602945 281473433893568 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    I0801 05:53:16.626557 281473433893568 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    I0801 05:53:16.641208 281473433893568 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:16.642487 281473433893568 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    I0801 05:53:16.642794 281473433893568 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    W0801 05:53:16.643112 281473433893568 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    INFO:tensorflow:Start training at 0
[worker-0]:    I0801 05:53:16.643298 281473433893568 failure_handler_test.py:197] Start training at 0
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    I0801 05:53:16.647869 281473433893568 failure_handling.py:634] Start watcher for peer's signal.
[worker-2]:    I0801 05:53:16.636223 281473433893568 failure_handling.py:674] Start watcher for local signal.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    I0801 05:53:16.636567 281473433893568 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    W0801 05:53:16.636873 281473433893568 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    INFO:tensorflow:Start training at 0
[worker-2]:    I0801 05:53:16.637058 281473433893568 failure_handler_test.py:197] Start training at 0
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-3]:    I0801 05:53:16.666505 281473433893568 failure_handling.py:674] Start watcher for local signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-3]:    I0801 05:53:16.666838 281473433893568 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    W0801 05:53:16.667163 281473433893568 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start training at 0
[worker-3]:    I0801 05:53:16.667348 281473433893568 failure_handler_test.py:197] Start training at 0
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-1]:    I0801 05:53:16.676216 281473433893568 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-1]:    I0801 05:53:16.676543 281473433893568 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    W0801 05:53:16.676878 281473433893568 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    INFO:tensorflow:Start training at 0
[worker-1]:    I0801 05:53:16.677067 281473433893568 failure_handler_test.py:197] Start training at 0
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:16.907476 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:16.934109 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:16.953009 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:16.958767 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.047394 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.051040 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.060941 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:17.071132 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:17.167049 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.166898 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.181097 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.190628 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.298492 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.316442 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:17.325801 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.340525 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.425314 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.441264 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:17.437119 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.450722 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d9da0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:17.557941 281473433893568 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d9da0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d5620> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d8a40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9da340> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:17.563723 281473433893568 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d8a40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:17.563740 281473433893568 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9da340> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:17.558297 281473433893568 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d5620> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:17.567297 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.580607 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.590641 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.600595 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9da660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:17.727990 281473433893568 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9da660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d6200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:17.733997 281473433893568 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d6200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9db560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:17.734307 281473433893568 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9db560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d9760> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:17.736367 281473433893568 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff9f9d9760> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.739506 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:17.742846 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.745217 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.744999 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.844737 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.833371 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:17.855719 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.855947 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:17.978832 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:17.983742 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:17.970034 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:18.000488 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:18.080530 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:18.096629 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:18.089815 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:18.117857 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:18.255836 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:18.271377 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:18.290342 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:18.310581 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:18.835950 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:18.837196 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:18.856506 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:18.861315 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:18.945905 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:18.952744 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:18.953004 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:18.970511 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.068351 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.069321 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.069885 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.069376 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.138372 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.136726 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.160543 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.160497 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 0 finished
[worker-3]:    I0801 05:53:19.261015 281473433893568 failure_handler_test.py:195] epoch 0 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.271791 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 0 finished
[worker-1]:    INFO:tensorflow:epoch 0 finished
[worker-1]:    I0801 05:53:19.278607 281473433893568 failure_handler_test.py:195] epoch 0 finished
[worker-0]:    I0801 05:53:19.276896 281473433893568 failure_handler_test.py:195] epoch 0 finished
[worker-2]:    INFO:tensorflow:epoch 0 finished
[worker-2]:    I0801 05:53:19.279376 281473433893568 failure_handler_test.py:195] epoch 0 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.394493 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.428834 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.447309 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.601899 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.609731 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.606897 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.630796 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.703655 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.708953 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.720473 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.720565 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.799267 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.799805 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.799559 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.799736 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.888365 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.890397 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.900567 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.907099 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.973581 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.975324 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.994522 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.981203 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.072338 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.072198 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.071809 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.072823 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.143247 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.146173 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.151597 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.147337 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.240445 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.230269 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.240534 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.250403 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.530914 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.525561 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.541645 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.585901 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.699619 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.710742 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.726375 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.760375 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.955003 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.986644 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.955082 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.985432 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.045566 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.045978 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.049118 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.049118 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.104927 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.104234 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.105205 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.172750 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.107501 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.173558 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.173607 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.172763 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    INFO:tensorflow:epoch 1 finished
[worker-3]:    I0801 05:53:21.300876 281473433893568 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    I0801 05:53:21.301251 281473433893568 failure_handler_test.py:195] epoch 1 finished
[worker-2]:    INFO:tensorflow:epoch 1 finished
[worker-1]:    I0801 05:53:21.301504 281473433893568 failure_handler_test.py:195] epoch 1 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.310427 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.304703 281473433893568 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.315891 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.336460 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.337373 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.401676 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.402985 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.400953 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.420523 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.557558 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.565194 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.568848 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.610348 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.755482 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.756397 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.755498 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.756375 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.859402 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.864639 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.883833 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.895033 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.044175 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.045425 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.061378 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.072257 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.131141 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.131253 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.130898 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.130110 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.191800 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.192019 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.211367 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.211706 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.271361 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.286076 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.286123 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.298997 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.381700 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.412686 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.438472 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.460543 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.554126 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.556654 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.591493 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.590606 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.680207 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.681420 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.686350 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.701305 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.763639 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.764182 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.783752 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.780553 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.896597 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.896892 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.906090 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.915407 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.996868 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.002863 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.004700 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.014851 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 2 finished
[worker-0]:    INFO:tensorflow:epoch 2 finished
[worker-3]:    I0801 05:53:23.093353 281473433893568 failure_handler_test.py:195] epoch 2 finished
[worker-0]:    I0801 05:53:23.093302 281473433893568 failure_handler_test.py:195] epoch 2 finished
[worker-1]:    INFO:tensorflow:epoch 2 finished
[worker-1]:    I0801 05:53:23.096509 281473433893568 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    INFO:tensorflow:epoch 2 finished
[worker-2]:    I0801 05:53:23.100568 281473433893568 failure_handler_test.py:195] epoch 2 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.104506 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.109868 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.116276 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.116344 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.206235 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.204858 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.202929 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.224138 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.338038 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.338137 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.351187 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.360432 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:sending SIGTERM
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:23.657416 281472910916288 failure_handler_test.py:438] sending SIGTERM
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.459657 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.457543 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[  FAILED  ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker): 28.52s
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:39.351677 281472910916288 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker): 28.52s
[ RUN      ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSlocal
[worker-2]:    I0801 05:53:23.461567 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.457543 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.518212 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.519520 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.519349 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.518361 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.579489 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.580707 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.580597 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.579369 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.641126 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.641540 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.639881 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.641015 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.701494 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.701100 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.700397 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.699592 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.759104 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.759016 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.758742 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.757943 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.817414 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.817229 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.818047 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.816092 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.877157 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.877152 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.876821 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.876688 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.950473 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.935304 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.934961 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.933771 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.007837 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.012804 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.007689 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.008291 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.067816 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.068966 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.069224 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.129697 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.147133 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.129783 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.069184 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 3 finished
[worker-3]:    INFO:tensorflow:epoch 3 finished
[worker-2]:    INFO:tensorflow:epoch 3 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.195277 281473433893568 failure_handler_test.py:195] epoch 3 finished
[worker-3]:    I0801 05:53:24.194814 281473433893568 failure_handler_test.py:195] epoch 3 finished
[worker-2]:    I0801 05:53:24.195264 281473433893568 failure_handler_test.py:195] epoch 3 finished
[worker-0]:    I0801 05:53:24.130153 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 3 finished
[worker-1]:    I0801 05:53:24.205447 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.203977 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.204598 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.195086 281473433893568 failure_handler_test.py:195] epoch 3 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.263110 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.262937 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.205159 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.263929 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.321176 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.261142 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.320928 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.321224 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.378585 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.320951 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.377867 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.435324 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.377397 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.492914 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.435412 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.376732 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.435356 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.550274 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.493342 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.434046 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.491556 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.607022 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.549842 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.491280 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:39.432966 281472910916288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',)
INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
I0801 05:53:39.433665 281472910916288 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:24.548631 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.662921 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.548407 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.606838 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.605863 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.719944 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.606708 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.663917 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.662171 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.662038 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.775326 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.719161 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.720121 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.718972 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.833600 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.775479 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.776204 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.774580 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.889415 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.833004 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.833271 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.833803 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.946003 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.888710 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.888425 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.890551 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.001255 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.944950 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 4 finished
[worker-1]:    I0801 05:53:24.945686 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.945051 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.049263 281473433893568 failure_handler_test.py:195] epoch 4 finished
INFO:tensorflow:Start watcher for local signal.
I0801 05:53:39.461409 281472910916288 failure_handling.py:674] Start watcher for local signal.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
I0801 05:53:39.461956 281472910916288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    I0801 05:53:25.001486 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
Instructions for updating:
Track steps using a tf.Variable saved in checkpoint instead.
W0801 05:53:39.462502 281472910916288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
Instructions for updating:
Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:Start training at 0
I0801 05:53:39.462865 281472910916288 failure_handler_test.py:197] Start training at 0
[worker-1]:    I0801 05:53:25.002087 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 4 finished
[worker-2]:    I0801 05:53:25.001679 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.057743 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.049388 281473433893568 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    INFO:tensorflow:epoch 4 finished
[worker-2]:    INFO:tensorflow:epoch 4 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.049557 281473433893568 failure_handler_test.py:195] epoch 4 finished
[worker-3]:    I0801 05:53:25.113034 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.049579 281473433893568 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.057515 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.058662 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.057826 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.166941 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.112938 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.112556 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.112885 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.221252 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.276167 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.167109 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.167126 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.166784 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.330508 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.221180 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.221414 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.221465 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.383947 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.275970 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.276256 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.436990 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.276350 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.330372 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.330584 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.330612 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.492558 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.383951 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.384092 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.545861 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.384256 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.437108 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.437000 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.598909 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.437282 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.493116 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.492540 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.493212 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.651991 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.546303 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.545977 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.546067 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.704870 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.599166 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.757765 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.598905 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.599522 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.652273 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.652144 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.652389 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.811120 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.705037 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.704968 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 5 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.857302 281473433893568 failure_handler_test.py:195] epoch 5 finished
[worker-0]:    I0801 05:53:25.704857 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.757983 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.757812 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.865210 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.811323 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.918498 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.972853 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 5 finished
[worker-3]:    I0801 05:53:26.026706 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.811099 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.757891 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.857695 281473433893568 failure_handler_test.py:195] epoch 5 finished
[worker-0]:    INFO:tensorflow:epoch 5 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.865469 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.857454 281473433893568 failure_handler_test.py:195] epoch 5 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.080581 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.918750 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.811188 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.135149 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.865183 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 5 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.857653 281473433893568 failure_handler_test.py:195] epoch 5 finished
[worker-3]:    I0801 05:53:26.187703 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.973068 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.026537 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.080777 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.135470 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.918528 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.865340 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.972816 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.241854 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.918639 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.026411 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.972952 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.294879 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.080573 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.026451 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.347780 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.135308 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.080693 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.135434 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.187906 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.240777 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.294676 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.347814 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.402566 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.455337 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.507899 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.560579 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.612617 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.402394 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 6 finished
[worker-0]:    I0801 05:53:26.187815 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.658105 281473433893568 failure_handler_test.py:195] epoch 6 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.455206 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.240666 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.665663 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.507874 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.294592 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.720280 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.560461 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.347751 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.775426 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.187948 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.827477 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.879996 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.931970 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.983898 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.035717 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.087544 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.139457 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.191218 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.242945 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.612421 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.294655 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 6 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.346488 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.400186 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 7 finished
[worker-1]:    I0801 05:53:27.445233 281473433893568 failure_handler_test.py:195] epoch 7 finished
[worker-1]:    INFO:tensorflow:Training finished.
[worker-1]:    I0801 05:53:27.446326 281473433893568 failure_handler_test.py:245] Training finished.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.240836 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.294810 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.347848 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.657795 281473433893568 failure_handler_test.py:195] epoch 6 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.665463 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.402602 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.718048 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.455406 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.775086 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.507980 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.402900 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.560616 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.454832 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.612634 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.507549 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 6 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.658163 281473433893568 failure_handler_test.py:195] epoch 6 finished
[worker-0]:    I0801 05:53:26.560265 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.665798 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.827268 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.612267 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.718803 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.879632 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 6 finished
[worker-2]:    I0801 05:53:26.775440 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.657927 281473433893568 failure_handler_test.py:195] epoch 6 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.931780 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.665249 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.983675 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.827558 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.035506 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.717955 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.087362 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.775793 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.827274 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.139280 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.879958 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.880413 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.190988 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.931756 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.242713 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.984316 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.294544 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.932100 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.035471 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.346260 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.983993 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.088028 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.399914 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 7 finished
[worker-0]:    I0801 05:53:27.139328 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.035905 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.087718 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.191663 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.139633 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.242712 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.444929 281473433893568 failure_handler_test.py:195] epoch 7 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Training finished.
[worker-0]:    I0801 05:53:27.295173 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.445755 281473433893568 failure_handler_test.py:245] Training finished.
WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xfffed63c2ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
W0801 05:53:39.979560 281472910916288 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xfffed63c2ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    I0801 05:53:27.191359 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.243132 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.294902 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.346677 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.400356 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 7 finished
[worker-2]:    I0801 05:53:27.445287 281473433893568 failure_handler_test.py:195] epoch 7 finished
[worker-2]:    INFO:tensorflow:Training finished.
[worker-2]:    I0801 05:53:27.446420 281473433893568 failure_handler_test.py:245] Training finished.
[worker-0]:    I0801 05:53:27.346343 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.400041 281473433893568 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 7 finished
[worker-0]:    I0801 05:53:27.445062 281473433893568 failure_handler_test.py:195] epoch 7 finished
[worker-0]:    INFO:tensorflow:Training finished.
[worker-0]:    I0801 05:53:27.446161 281473433893568 failure_handler_test.py:245] Training finished.
WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xfffed63c0040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
W0801 05:53:40.029026 281472910916288 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xfffed63c0040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
INFO:tensorflow:epoch 0 finished
I0801 05:53:40.263450 281472910916288 failure_handler_test.py:195] epoch 0 finished
INFO:tensorflow:epoch 1 finished
I0801 05:53:43.016434 281472910916288 failure_handler_test.py:195] epoch 1 finished
INFO:tensorflow:epoch 2 finished
I0801 05:53:43.551142 281472910916288 failure_handler_test.py:195] epoch 2 finished
INFO:tensorflow:epoch 3 finished
I0801 05:53:44.059366 281472910916288 failure_handler_test.py:195] epoch 3 finished
INFO:tensorflow:epoch 4 finished
I0801 05:53:44.743933 281472910916288 failure_handler_test.py:195] epoch 4 finished
INFO:tensorflow:epoch 5 finished
I0801 05:53:45.213023 281472910916288 failure_handler_test.py:195] epoch 5 finished
INFO:tensorflow:sending sigterm
I0801 05:53:45.370717 281469971657184 failure_handler_test.py:327] sending sigterm
INFO:tensorflow:Member single_worker has received termination notice.
I0801 05:53:45.373344 281472910916288 failure_handling.py:701] Member single_worker has received termination notice.
INFO:tensorflow:Termination caught in main thread on preempted worker
I0801 05:53:45.382911 281472910916288 failure_handling.py:1159] Termination caught in main thread on preempted worker
INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
I0801 05:53:45.410888 281472910916288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/f32f67050ce4501934b14426193a0f197b8_vvk5/tmpyzz9wilb/fh_ckpt
I0801 05:53:45.507231 281472910916288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/f32f67050ce4501934b14426193a0f197b8_vvk5/tmpyzz9wilb/fh_ckpt
INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
I0801 05:53:45.507543 281472910916288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
INFO:tensorflow:restarting workers
I0801 05:53:45.507869 281472910916288 failure_handler_test.py:351] restarting workers
INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',)
I0801 05:53:45.511072 281472910916288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',)
INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
I0801 05:53:45.511397 281472910916288 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
INFO:tensorflow:Start watcher for local signal.
I0801 05:53:45.537823 281472910916288 failure_handling.py:674] Start watcher for local signal.
INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
I0801 05:53:45.538330 281472910916288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
INFO:tensorflow:Start training at 97
I0801 05:53:45.538593 281472910916288 failure_handler_test.py:197] Start training at 97
INFO:tensorflow:training restarted
I0801 05:53:45.538787 281472910916288 failure_handler_test.py:207] training restarted
INFO:tensorflow:epoch 6 finished
I0801 05:53:45.863890 281472910916288 failure_handler_test.py:195] epoch 6 finished
INFO:tensorflow:epoch 7 finished
I0801 05:53:46.386578 281472910916288 failure_handler_test.py:195] epoch 7 finished
INFO:tensorflow:Training finished.
I0801 05:53:46.386927 281472910916288 failure_handler_test.py:245] Training finished.
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSlocal): 7.04s
I0801 05:53:46.388693 281472910916288 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSlocal): 7.04s
[       OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSlocal
[ RUN      ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSlocal
INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',)
I0801 05:53:46.410661 281472910916288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',)
INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
I0801 05:53:46.411035 281472910916288 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
INFO:tensorflow:Start watcher for local signal.
I0801 05:53:46.426626 281472910916288 failure_handling.py:674] Start watcher for local signal.
INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
I0801 05:53:46.427113 281472910916288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
INFO:tensorflow:Start training at 0
I0801 05:53:46.427364 281472910916288 failure_handler_test.py:197] Start training at 0
INFO:tensorflow:epoch 0 finished
I0801 05:53:46.958030 281472910916288 failure_handler_test.py:195] epoch 0 finished
INFO:tensorflow:epoch 1 finished
I0801 05:53:47.708813 281472910916288 failure_handler_test.py:195] epoch 1 finished
INFO:tensorflow:epoch 2 finished
I0801 05:53:48.233416 281472910916288 failure_handler_test.py:195] epoch 2 finished
INFO:tensorflow:epoch 3 finished
I0801 05:53:48.683209 281472910916288 failure_handler_test.py:195] epoch 3 finished
INFO:tensorflow:epoch 4 finished
I0801 05:53:49.292983 281472910916288 failure_handler_test.py:195] epoch 4 finished
INFO:tensorflow:epoch 5 finished
I0801 05:53:49.795379 281472910916288 failure_handler_test.py:195] epoch 5 finished
INFO:tensorflow:epoch 6 finished
I0801 05:53:50.278429 281472910916288 failure_handler_test.py:195] epoch 6 finished
INFO:tensorflow:sending sigterm
I0801 05:53:50.416521 281469971657184 failure_handler_test.py:327] sending sigterm
INFO:tensorflow:Member single_worker has received termination notice.
I0801 05:53:50.425730 281472910916288 failure_handling.py:701] Member single_worker has received termination notice.
INFO:tensorflow:Termination caught in main thread on preempted worker
I0801 05:53:50.426340 281472910916288 failure_handling.py:1159] Termination caught in main thread on preempted worker
INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
I0801 05:53:50.453703 281472910916288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/f32f67050ce4501934b14426193a0f197b8_vvk5/tmp95at37hg/fh_ckpt
I0801 05:53:50.526639 281472910916288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/f32f67050ce4501934b14426193a0f197b8_vvk5/tmp95at37hg/fh_ckpt
INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
I0801 05:53:50.526958 281472910916288 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
INFO:tensorflow:restarting workers
I0801 05:53:50.527286 281472910916288 failure_handler_test.py:351] restarting workers
INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',)
I0801 05:53:50.530323 281472910916288 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',)
INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
I0801 05:53:50.530665 281472910916288 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO
INFO:tensorflow:Start watcher for local signal.
I0801 05:53:50.561203 281472910916288 failure_handling.py:674] Start watcher for local signal.
INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
I0801 05:53:50.561693 281472910916288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
INFO:tensorflow:Start training at 111
I0801 05:53:50.561955 281472910916288 failure_handler_test.py:197] Start training at 111
INFO:tensorflow:training restarted
I0801 05:53:50.562153 281472910916288 failure_handler_test.py:207] training restarted
INFO:tensorflow:epoch 7 finished
I0801 05:53:50.951966 281472910916288 failure_handler_test.py:195] epoch 7 finished
INFO:tensorflow:Training finished.
I0801 05:53:50.952561 281472910916288 failure_handler_test.py:245] Training finished.
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSlocal): 4.56s
I0801 05:53:50.954875 281472910916288 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSlocal): 4.56s
[       OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSlocal
======================================================================
ERROR: test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest)
PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker
test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_MWMSmultiworker(input_arg='checkpoint', strategy_option='MWMS_multi_worker')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test
    return test_method(self, **testcase_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated
    execute_test_method()
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator
    test_method(self, **kwargs)
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 439, in test_grace_period_continue_training
    os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM)
ProcessLookupError: [Errno 3] No such process

----------------------------------------------------------------------
Ran 3 tests in 40.127s

FAILED (errors=1)
================================================================================
==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 5 of 8):
2023-08-01 05:53:02.358901: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Running tests under Python 3.11.1: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3
[ RUN      ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice
INFO:tensorflow:Start watcher for local signal.
I0801 05:53:11.021408 281473864792768 failure_handling.py:674] Start watcher for local signal.
INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
I0801 05:53:11.021894 281473864792768 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
Instructions for updating:
Track steps using a tf.Variable saved in checkpoint instead.
W0801 05:53:11.022230 281473864792768 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
Instructions for updating:
Track steps using a tf.Variable saved in checkpoint instead.
INFO:tensorflow:Start training at 0
I0801 05:53:11.022452 281473864792768 failure_handler_test.py:197] Start training at 0
WARNING:tensorflow:5 out of the last 5 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xffff0f302d40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
W0801 05:53:11.551864 281473864792768 polymorphic_function.py:156] 5 out of the last 5 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xffff0f302d40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xffff0f302d40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
W0801 05:53:11.587337 281473864792768 polymorphic_function.py:156] 6 out of the last 6 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xffff0f302d40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
INFO:tensorflow:epoch 0 finished
I0801 05:53:12.037212 281473864792768 failure_handler_test.py:195] epoch 0 finished
INFO:tensorflow:epoch 1 finished
I0801 05:53:12.335755 281473864792768 failure_handler_test.py:195] epoch 1 finished
INFO:tensorflow:epoch 2 finished
I0801 05:53:12.566859 281473864792768 failure_handler_test.py:195] epoch 2 finished
INFO:tensorflow:epoch 3 finished
I0801 05:53:12.924848 281473864792768 failure_handler_test.py:195] epoch 3 finished
INFO:tensorflow:epoch 4 finished
I0801 05:53:13.220743 281473864792768 failure_handler_test.py:195] epoch 4 finished
INFO:tensorflow:epoch 5 finished
I0801 05:53:13.489037 281473864792768 failure_handler_test.py:195] epoch 5 finished
INFO:tensorflow:epoch 6 finished
I0801 05:53:13.782857 281473864792768 failure_handler_test.py:195] epoch 6 finished
INFO:tensorflow:sending sigterm
I0801 05:53:13.876213 281470934577632 failure_handler_test.py:467] sending sigterm
INFO:tensorflow:Member single_worker has received termination notice.
I0801 05:53:13.877786 281473864792768 failure_handling.py:701] Member single_worker has received termination notice.
INFO:tensorflow:Termination caught in main thread on preempted worker
I0801 05:53:13.889077 281473864792768 failure_handling.py:1159] Termination caught in main thread on preempted worker
INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
I0801 05:53:13.917207 281473864792768 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpdbrez9mc/fh_ckpt
I0801 05:53:14.000734 281473864792768 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpdbrez9mc/fh_ckpt
INFO:tensorflow:Continue training for the grace period.
I0801 05:53:14.001075 281473864792768 failure_handling.py:1134] Continue training for the grace period.
INFO:tensorflow:epoch 7 finished
I0801 05:53:14.100589 281473864792768 failure_handler_test.py:195] epoch 7 finished
INFO:tensorflow:Training finished.
I0801 05:53:14.101211 281473864792768 failure_handler_test.py:245] Training finished.
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 3.27s
I0801 05:53:14.102439 281473864792768 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 3.27s
[       OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice
[ RUN      ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker
INFO:tensorflow:Using local port 39489
I0801 05:53:14.107397 281473864792768 test_util.py:3813] Using local port 39489
INFO:tensorflow:Using local port 38009
I0801 05:53:14.107876 281473864792768 test_util.py:3813] Using local port 38009
INFO:tensorflow:Using local port 36675
I0801 05:53:14.108231 281473864792768 test_util.py:3813] Using local port 36675
INFO:tensorflow:Using local port 33193
I0801 05:53:14.108576 281473864792768 test_util.py:3813] Using local port 33193
2023-08-01 05:53:15.517950: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-01 05:53:15.523736: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
INFO:tensorflow:Cluster starting.
I0801 05:53:18.668455 281473864792768 failure_handler_test.py:297] Cluster starting.
[worker-0]:    I0801 05:53:18.748157 281473656060608 multi_process_runner.py:840] Subprocess with PID 2924211 (worker, 0) is now being started.
[worker-0]:    I0801 05:53:18.748541 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-0]:    2023-08-01 05:53:18.810979: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39489
[worker-0]:    2023-08-01 05:53:18.848088: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 15432973044908955675
[worker-0]:    2023-08-01 05:53:18.848479: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    I0801 05:53:18.868155 281473656060608 multi_process_runner.py:840] Subprocess with PID 2924274 (worker, 2) is now being started.
[worker-1]:    I0801 05:53:18.886261 281473656060608 multi_process_runner.py:840] Subprocess with PID 2924259 (worker, 1) is now being started.
[worker-2]:    I0801 05:53:18.868545 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:18.897761 281473656060608 multi_process_runner.py:840] Subprocess with PID 2924301 (worker, 3) is now being started.
[worker-3]:    I0801 05:53:18.898164 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-1]:    I0801 05:53:18.886648 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-3]:    2023-08-01 05:53:18.993198: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33193
[worker-1]:    2023-08-01 05:53:19.026953: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38009
[worker-3]:    2023-08-01 05:53:19.027593: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    2023-08-01 05:53:19.028735: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:36675
[worker-0]:    2023-08-01 05:53:19.027169: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 1823841893677072606
[worker-1]:    2023-08-01 05:53:19.046891: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:19.046609: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 17409505414429458987
[worker-0]:    2023-08-01 05:53:19.096184: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 14301748006310533934
[worker-2]:    2023-08-01 05:53:19.096688: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    I0801 05:53:19.100813 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-3]:    I0801 05:53:19.102628 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:19.103436 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    I0801 05:53:19.100454 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    I0801 05:53:19.170894 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:19.171513 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:19.172346 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-1]:    I0801 05:53:19.171755 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    I0801 05:53:19.173063 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:19.173315 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    I0801 05:53:19.175005 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    I0801 05:53:19.175564 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:19.175797 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    I0801 05:53:19.178129 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:19.178848 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:19.179094 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-2]:    I0801 05:53:19.274521 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    I0801 05:53:19.276962 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-1]:    I0801 05:53:19.289348 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    I0801 05:53:19.293806 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-3]:    I0801 05:53:19.294964 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-3]:    I0801 05:53:19.295270 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    W0801 05:53:19.295620 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start training at 0
[worker-3]:    I0801 05:53:19.295813 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-1]:    I0801 05:53:19.347551 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-1]:    I0801 05:53:19.347897 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    W0801 05:53:19.348204 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    INFO:tensorflow:Start training at 0
[worker-1]:    I0801 05:53:19.348395 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-2]:    I0801 05:53:19.355558 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    I0801 05:53:19.355880 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    W0801 05:53:19.356207 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    INFO:tensorflow:Start training at 0
[worker-2]:    I0801 05:53:19.356391 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:19.406987 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    I0801 05:53:19.407322 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    W0801 05:53:19.407636 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    INFO:tensorflow:Start training at 0
[worker-0]:    I0801 05:53:19.407832 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.559547 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.540883 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.579401 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.621703 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.705348 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.705751 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.705826 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.723594 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.825466 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:19.825671 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.831148 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.841433 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:19.960280 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:19.982355 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:19.971510 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.020639 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.241773 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.246305 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.251421 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.257242 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbce0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:20.319722 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbce0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbb920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:20.327433 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbb920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.329284 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:20.327092 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.337199 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.367331 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbce0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:20.366691 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbce0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.376980 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbc40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:20.508517 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbc40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba3e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:20.515059 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba3e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba3e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba520> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:20.515479 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba3e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:20.515446 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdba520> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.524718 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.526016 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.518272 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.541484 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.646693 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.652736 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.652763 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.671362 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.736050 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.736717 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.736659 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.761256 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.827618 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.841137 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.826861 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.835725 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:20.910309 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:20.910462 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:20.911612 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:20.910684 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.116574 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.174494 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.116554 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.116746 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.174238 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.176155 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.117975 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.175060 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.235479 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.235677 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.236035 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.235600 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.308580 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.308709 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.310359 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.306960 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 0 finished
[worker-3]:    I0801 05:53:21.387880 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-0]:    INFO:tensorflow:epoch 0 finished
[worker-0]:    I0801 05:53:21.389810 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-1]:    INFO:tensorflow:epoch 0 finished
[worker-2]:    INFO:tensorflow:epoch 0 finished
[worker-1]:    I0801 05:53:21.391141 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-2]:    I0801 05:53:21.391424 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.400650 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.401495 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.401268 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.405373 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.464857 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.465551 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.465554 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.472550 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.534005 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.533562 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.539009 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.555389 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.650237 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.650312 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.650605 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.680904 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.757632 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.757741 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.758330 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.757224 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:sending sigterm
I0801 05:53:21.783689 281473864792768 failure_handler_test.py:302] sending sigterm
INFO:tensorflow:sigterm sent
I0801 05:53:21.784096 281473864792768 failure_handler_test.py:306] sigterm sent
[worker-2]:    INFO:tensorflow:Member 2 has received termination notice.
[worker-2]:    I0801 05:53:21.785058 281473656060608 failure_handling.py:710] Member 2 has received termination notice.
[worker-2]:    INFO:tensorflow:Termination caught in main thread on preempted worker
[worker-2]:    I0801 05:53:21.812655 281473656060608 failure_handling.py:1159] Termination caught in main thread on preempted worker
[worker-2]:    INFO:tensorflow:RUN_TO_CHECKPOINT set to 21
[worker-2]:    I0801 05:53:21.813665 281473656060608 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 21
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 0 received
[worker-2]:    I0801 05:53:21.816499 281473656060608 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged
[worker-3]:    I0801 05:53:21.816981 281448359326176 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 1 received
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged
[worker-2]:    I0801 05:53:21.817731 281473656060608 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received
[worker-1]:    I0801 05:53:21.818071 281448242016736 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged
[worker-0]:    I0801 05:53:21.817422 281447554019808 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:21.822719 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 2 received
[worker-2]:    I0801 05:53:21.827355 281473656060608 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged
[worker-2]:    I0801 05:53:21.827578 281447193637344 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 3 received
[worker-2]:    I0801 05:53:21.828584 281473656060608 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.823422 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:21.822463 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:21.838409 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-3]:    I0801 05:53:21.892091 281473656060608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-0]:    I0801 05:53:21.892240 281473656060608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-2]:    I0801 05:53:21.894417 281473656060608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-1]:    I0801 05:53:21.894238 281473656060608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-0]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/fh_ckpt
[worker-3]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/workertemp_3/fh_ckpt
[worker-0]:    I0801 05:53:21.952040 281473656060608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/fh_ckpt
[worker-3]:    I0801 05:53:21.952096 281473656060608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/workertemp_3/fh_ckpt
[worker-2]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/workertemp_2/fh_ckpt
[worker-3]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-2]:    I0801 05:53:21.953969 281473656060608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/workertemp_2/fh_ckpt
[worker-0]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-3]:    I0801 05:53:21.954055 281473656060608 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-0]:    I0801 05:53:21.954349 281473656060608 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-3]:    I0801 05:53:21.954264 281473656060608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-0]:    I0801 05:53:21.954577 281473656060608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-2]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-2]:    I0801 05:53:21.955950 281473656060608 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-2]:    I0801 05:53:21.956194 281473656060608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-1]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/workertemp_1/fh_ckpt
[worker-1]:    I0801 05:53:21.968624 281473656060608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad227n2m927/tmpup2k6sz4/workertemp_1/fh_ckpt
[worker-1]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-1]:    I0801 05:53:21.970763 281473656060608 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-1]:    I0801 05:53:21.971062 281473656060608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
INFO:tensorflow:restarting workers
I0801 05:53:23.786405 281473864792768 failure_handler_test.py:309] restarting workers
[worker-0]:    I0801 05:53:39.386666 281473656060608 multi_process_runner.py:840] Subprocess with PID 2935369 (worker, 0) is now being started.
[worker-0]:    I0801 05:53:39.387151 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
INFO:tensorflow:workers restarted
I0801 05:53:39.395613 281473864792768 failure_handler_test.py:313] workers restarted
[worker-1]:    I0801 05:53:39.441952 281473656060608 multi_process_runner.py:840] Subprocess with PID 2935557 (worker, 1) is now being started.
[worker-1]:    I0801 05:53:39.442347 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:39.454670 281473656060608 multi_process_runner.py:840] Subprocess with PID 2935632 (worker, 3) is now being started.
[worker-2]:    I0801 05:53:39.454782 281473656060608 multi_process_runner.py:840] Subprocess with PID 2935589 (worker, 2) is now being started.
[worker-2]:    I0801 05:53:39.455206 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:39.455171 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:39489", "localhost:38009", "localhost:36675", "localhost:33193"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-1]:    2023-08-01 05:53:39.485496: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38009
[worker-3]:    2023-08-01 05:53:39.503611: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33193
[worker-2]:    2023-08-01 05:53:39.504099: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:36675
[worker-0]:    2023-08-01 05:53:39.508410: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39489
[worker-0]:    2023-08-01 05:53:39.509609: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 15796615643544742300
[worker-1]:    2023-08-01 05:53:39.509941: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:39.516581: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 11115795027081178684
[worker-2]:    2023-08-01 05:53:39.516876: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:39.542196: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 10085995842571850277
[worker-0]:    2023-08-01 05:53:39.542415: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:39.543186: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 5659024713182289895
[worker-3]:    2023-08-01 05:53:39.543389: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:39.545209 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    I0801 05:53:39.545254 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-1]:    I0801 05:53:39.545504 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    I0801 05:53:39.545336 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-3]:    I0801 05:53:39.608163 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-0]:    I0801 05:53:39.608164 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:39.608712 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    I0801 05:53:39.608715 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:39.608950 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:39.608954 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    I0801 05:53:39.608162 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    I0801 05:53:39.608712 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:39.608949 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    I0801 05:53:39.649065 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:39.649816 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:39.650063 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:39489', 'localhost:38009', 'localhost:36675', 'localhost:33193']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-2]:    I0801 05:53:39.742238 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    I0801 05:53:39.742221 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-2]:    I0801 05:53:39.743274 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:39.747443 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    I0801 05:53:39.747773 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-3]:    I0801 05:53:39.748774 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-1]:    I0801 05:53:39.748780 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-3]:    I0801 05:53:39.749717 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    W0801 05:53:39.748092 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    I0801 05:53:39.749764 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    Instructions for updating:
[worker-3]:    I0801 05:53:39.749987 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    I0801 05:53:39.750028 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-3]:    Instructions for updating:
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    INFO:tensorflow:Start training at 21
[worker-0]:    I0801 05:53:39.748287 281473656060608 failure_handler_test.py:197] Start training at 21
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-1]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    I0801 05:53:39.743557 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    INFO:tensorflow:training restarted
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    W0801 05:53:39.750292 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    I0801 05:53:39.760564 281473656060608 failure_handler_test.py:207] training restarted
[worker-3]:    Instructions for updating:
[worker-2]:    Instructions for updating:
[worker-1]:    W0801 05:53:39.750350 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    Instructions for updating:
[worker-3]:    INFO:tensorflow:Start training at 21
[worker-2]:    W0801 05:53:39.743861 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    I0801 05:53:39.750488 281473656060608 failure_handler_test.py:197] Start training at 21
[worker-2]:    Instructions for updating:
[worker-1]:    INFO:tensorflow:Start training at 21
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    I0801 05:53:39.750543 281473656060608 failure_handler_test.py:197] Start training at 21
[worker-2]:    INFO:tensorflow:Start training at 21
[worker-2]:    I0801 05:53:39.744053 281473656060608 failure_handler_test.py:197] Start training at 21
[worker-2]:    INFO:tensorflow:training restarted
[worker-2]:    I0801 05:53:39.746088 281473656060608 failure_handler_test.py:207] training restarted
[worker-3]:    INFO:tensorflow:training restarted
[worker-3]:    I0801 05:53:39.804461 281473656060608 failure_handler_test.py:207] training restarted
[worker-1]:    INFO:tensorflow:training restarted
[worker-1]:    I0801 05:53:39.807017 281473656060608 failure_handler_test.py:207] training restarted
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:39.925111 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:39.939285 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:39.988096 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:39.979150 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.076957 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.079065 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.077474 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.077705 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.139516 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.140641 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.139750 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.140681 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.199553 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.200423 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.200541 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.200465 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.259430 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.260135 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.259039 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc04a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc3d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    I0801 05:53:40.259292 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    W0801 05:53:40.309573 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc3d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc0e00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:40.309288 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc04a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc0400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:40.309487 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc0e00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    W0801 05:53:40.309079 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc0400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    I0801 05:53:40.318944 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.319150 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.318366 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc22a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc2e80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    I0801 05:53:40.318627 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc3560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:40.368294 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc22a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc2de0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:40.368103 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc3560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:40.367907 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc2e80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.377946 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    W0801 05:53:40.367709 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdc2de0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    I0801 05:53:40.376791 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.376487 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.435494 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.434817 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.376435 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.435549 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.435244 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.493408 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.493465 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    I0801 05:53:40.542304 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    I0801 05:53:40.493596 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 1 finished
[worker-1]:    I0801 05:53:40.542520 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.552560 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.551921 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.493997 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 1 finished
[worker-1]:    I0801 05:53:40.609143 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.609987 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.542565 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-0]:    I0801 05:53:40.669712 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.542168 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    I0801 05:53:40.670457 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.551256 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.872378 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.609631 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.872359 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.932848 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.552138 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.991348 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.932640 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.608631 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.050626 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.669939 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.669752 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.111227 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.875973 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.872776 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.932664 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.992407 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.051629 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.991341 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.171312 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.110277 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.050571 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.228812 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.111091 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.170335 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.287749 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.170294 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.229322 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.228084 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.347011 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.288459 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.287963 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.403743 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.345852 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.345757 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.461395 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.932910 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.992523 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.051720 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.112026 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.170974 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.230101 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.288890 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.346302 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.403634 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.403728 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.403662 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.461932 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.518892 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.461235 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.461273 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 2 finished
[worker-2]:    I0801 05:53:41.517945 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.573216 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    INFO:tensorflow:epoch 2 finished
[worker-1]:    I0801 05:53:41.519335 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.518954 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 2 finished
[worker-2]:    I0801 05:53:41.573472 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-3]:    INFO:tensorflow:epoch 2 finished
[worker-1]:    I0801 05:53:41.573442 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-0]:    I0801 05:53:41.582523 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.573041 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.582619 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.583273 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.646508 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.582562 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.647198 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.647264 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.704227 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.647106 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.704193 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.705328 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.762706 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.762625 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.705095 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.763134 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.819565 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.820278 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.762913 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.818971 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.877549 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.877885 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.819883 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.876803 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.935575 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.934882 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.878209 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.934591 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.993231 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.992496 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.934843 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.992428 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.051820 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.050197 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.993435 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.050485 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.108658 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.108332 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.050130 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.108292 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.165625 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.165630 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.107997 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.169471 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.225655 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.225267 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.169469 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.225359 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.285121 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.285245 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.225083 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.284255 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.343147 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.343064 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.285235 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.342154 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.409908 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.342760 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.404681 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.406707 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.403121 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 3 finished
[worker-2]:    INFO:tensorflow:epoch 3 finished
[worker-3]:    I0801 05:53:42.504382 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.514997 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 3 finished
[worker-2]:    I0801 05:53:42.505410 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-1]:    INFO:tensorflow:epoch 3 finished
[worker-1]:    I0801 05:53:42.516709 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-0]:    I0801 05:53:42.504637 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.514780 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.514976 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.537603 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.672559 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.675661 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.700605 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.675566 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.783366 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.796480 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.812583 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.811624 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.883649 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.884261 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.883842 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.884584 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.950997 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.955556 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.971460 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.972793 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.085580 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.088583 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.089221 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.092007 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.167202 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.174454 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.174718 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.211327 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.288418 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.292853 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.292494 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.296589 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.396744 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.401490 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.391471 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.421470 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.496338 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.492533 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.506262 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.537405 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.615948 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.620655 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.631479 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.649118 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.738188 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.743883 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.750449 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.762649 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.856533 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.856564 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.861460 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.861036 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:43.970316 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:43.986598 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:43.992057 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:43.996464 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.087897 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.096154 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:44.088000 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.081628 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 4 finished
[worker-2]:    INFO:tensorflow:epoch 4 finished
[worker-1]:    INFO:tensorflow:epoch 4 finished
[worker-0]:    I0801 05:53:44.154727 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-2]:    I0801 05:53:44.154968 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    I0801 05:53:44.154879 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-3]:    INFO:tensorflow:epoch 4 finished
[worker-3]:    I0801 05:53:44.164477 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.170156 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:44.169291 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.170817 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.205949 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.293354 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:44.311645 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.301639 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.331650 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:44.427700 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.437148 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.461448 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.491393 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.589039 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:44.607939 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.616466 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.621076 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.737691 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:44.745662 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.751682 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.781575 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:44.856201 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.855908 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.856899 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.861668 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:44.971050 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:44.971141 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:44.991674 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.010733 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.107961 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.116385 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.111376 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.131270 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.199407 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.199701 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.203955 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.199828 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.298513 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.308427 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.311191 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.317259 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.438744 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.442822 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.443503 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.486289 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.568241 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.569350 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.568692 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.574803 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.660605 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.686642 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.681698 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.691662 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.823465 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.814126 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.841227 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.841129 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.967843 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:45.971671 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.987344 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:45.992493 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 5 finished
[worker-0]:    I0801 05:53:46.081321 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-2]:    INFO:tensorflow:epoch 5 finished
[worker-2]:    I0801 05:53:46.082452 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-3]:    INFO:tensorflow:epoch 5 finished
[worker-3]:    I0801 05:53:46.081160 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-1]:    INFO:tensorflow:epoch 5 finished
[worker-1]:    I0801 05:53:46.088847 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.093811 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.111529 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.104875 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.131641 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.212810 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.212824 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.212893 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.229657 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.326828 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.326926 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.326859 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.327196 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.404831 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.404999 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.407649 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.431482 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.526907 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.513043 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.524741 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.540607 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.657324 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.661218 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.665076 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.669469 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.752025 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.753169 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.757453 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.767212 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.836247 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.851319 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.846371 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.865262 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.939430 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.940015 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.940042 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.939762 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.011874 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.011869 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.013645 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.016104 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.100795 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.100714 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.100914 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.121543 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.191097 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.191100 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.190898 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.206315 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.317020 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.324599 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.325972 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.341095 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.459787 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.461435 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.456222 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.467006 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.531049 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.531989 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.531686 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.581387 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 6 finished
[worker-0]:    I0801 05:53:47.647745 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-3]:    INFO:tensorflow:epoch 6 finished
[worker-3]:    I0801 05:53:47.656783 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-2]:    INFO:tensorflow:epoch 6 finished
[worker-1]:    INFO:tensorflow:epoch 6 finished
[worker-2]:    I0801 05:53:47.662497 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-1]:    I0801 05:53:47.662377 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.673372 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.658635 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.677562 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.677098 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.782172 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.791389 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.791384 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.811098 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.876684 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.875992 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.891687 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.919215 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.016870 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.017017 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.016864 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.052347 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.120890 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.134794 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.142369 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.148689 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.247754 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.247660 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.286169 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.284006 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.397456 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.414106 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.415281 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.432077 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.522702 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.514640 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.536291 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.551360 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.739069 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.761008 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.763894 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.773085 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.889809 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.905268 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.910040 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.938566 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.023957 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.031683 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.032000 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.041571 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.128826 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.130155 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.137859 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.141507 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.246199 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.246226 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.262520 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.268116 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.398265 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.405775 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.417503 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.426188 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.546216 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.547983 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.546246 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.558636 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 7 finished
[worker-3]:    I0801 05:53:49.667649 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-3]:    INFO:tensorflow:Training finished.
[worker-3]:    I0801 05:53:49.669655 281473656060608 failure_handler_test.py:245] Training finished.
[worker-0]:    INFO:tensorflow:epoch 7 finished
[worker-0]:    I0801 05:53:49.676677 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-1]:    INFO:tensorflow:epoch 7 finished
[worker-1]:    I0801 05:53:49.678476 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-2]:    INFO:tensorflow:epoch 7 finished
[worker-2]:    I0801 05:53:49.679547 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-2]:    INFO:tensorflow:Training finished.
[worker-2]:    I0801 05:53:49.680804 281473656060608 failure_handler_test.py:245] Training finished.
[worker-0]:    INFO:tensorflow:Training finished.
[worker-0]:    I0801 05:53:49.678586 281473656060608 failure_handler_test.py:245] Training finished.
[worker-1]:    INFO:tensorflow:Training finished.
[worker-1]:    I0801 05:53:49.686736 281473656060608 failure_handler_test.py:245] Training finished.
I0801 05:53:50.422319 281473864792768 multi_process_runner.py:646] worker-0 exit code: 0
I0801 05:53:50.422644 281473864792768 multi_process_runner.py:646] worker-1 exit code: 0
I0801 05:53:50.422822 281473864792768 multi_process_runner.py:646] worker-2 exit code: 0
I0801 05:53:50.422989 281473864792768 multi_process_runner.py:646] worker-3 exit code: 0
I0801 05:53:50.426204 281473864792768 multi_process_runner.py:662] Joining log reading threads.
I0801 05:53:50.426546 281473864792768 multi_process_runner.py:665] Joined log reading threads.
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 36.63s
I0801 05:53:50.731956 281473864792768 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 36.63s
[       OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker
[ RUN      ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker
INFO:tensorflow:Using local port 46319
I0801 05:53:50.733626 281473864792768 test_util.py:3813] Using local port 46319
INFO:tensorflow:Using local port 36353
I0801 05:53:50.734009 281473864792768 test_util.py:3813] Using local port 36353
INFO:tensorflow:Using local port 41461
I0801 05:53:50.734367 281473864792768 test_util.py:3813] Using local port 41461
INFO:tensorflow:Using local port 35667
I0801 05:53:50.734724 281473864792768 test_util.py:3813] Using local port 35667
INFO:tensorflow:Cluster starting.
I0801 05:53:50.896652 281473864792768 failure_handler_test.py:297] Cluster starting.
[worker-1]:    I0801 05:53:51.028128 281473656060608 multi_process_runner.py:840] Subprocess with PID 2952827 (worker, 1) is now being started.
[worker-1]:    I0801 05:53:51.028562 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46319", "localhost:36353", "localhost:41461", "localhost:35667"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-2]:    I0801 05:53:51.099876 281473656060608 multi_process_runner.py:840] Subprocess with PID 2952960 (worker, 2) is now being started.
[worker-0]:    I0801 05:53:51.118034 281473656060608 multi_process_runner.py:840] Subprocess with PID 2952747 (worker, 0) is now being started.
[worker-2]:    I0801 05:53:51.100404 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46319", "localhost:36353", "localhost:41461", "localhost:35667"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:51.134164 281473656060608 multi_process_runner.py:840] Subprocess with PID 2952978 (worker, 3) is now being started.
[worker-1]:    2023-08-01 05:53:51.139863: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:36353
[worker-0]:    I0801 05:53:51.118510 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46319", "localhost:36353", "localhost:41461", "localhost:35667"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:51.134665 281473656060608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:46319", "localhost:36353", "localhost:41461", "localhost:35667"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-0]:    2023-08-01 05:53:51.200581: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:46319
[worker-0]:    2023-08-01 05:53:51.219624: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 9893816437333791491
[worker-0]:    2023-08-01 05:53:51.236429: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    2023-08-01 05:53:51.323578: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35667
[worker-0]:    2023-08-01 05:53:51.338646: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 12614318626557278754
[worker-3]:    2023-08-01 05:53:51.339823: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    2023-08-01 05:53:51.437641: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:41461
[worker-0]:    2023-08-01 05:53:51.440802: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 17349957395112981337
[worker-2]:    2023-08-01 05:53:51.441377: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:52.156318: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 16569416553009631652
[worker-1]:    2023-08-01 05:53:52.157509: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    I0801 05:53:52.169935 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    I0801 05:53:52.177993 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-3]:    I0801 05:53:52.171151 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:52.177967 281473656060608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    I0801 05:53:52.240005 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:52.240735 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:52.241001 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    I0801 05:53:52.243885 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:52.244571 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:52.244836 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    I0801 05:53:52.249166 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-0]:    I0801 05:53:52.245192 281473656060608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    I0801 05:53:52.249855 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    I0801 05:53:52.245877 281473656060608 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:52.250119 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:52.246164 281473656060608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:46319', 'localhost:36353', 'localhost:41461', 'localhost:35667']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-2]:    I0801 05:53:52.304332 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    I0801 05:53:52.304446 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    I0801 05:53:52.305509 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    I0801 05:53:52.306012 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-3]:    I0801 05:53:52.305799 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    W0801 05:53:52.306175 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start training at 0
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    I0801 05:53:52.306377 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-1]:    I0801 05:53:52.309824 281473656060608 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:52.313047 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:52.313426 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-2]:    I0801 05:53:52.313813 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-1]:    I0801 05:53:52.314939 281473656060608 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    Instructions for updating:
[worker-2]:    I0801 05:53:52.314504 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    I0801 05:53:52.315358 281473656060608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-2]:    Instructions for updating:
[worker-0]:    W0801 05:53:52.313784 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-2]:    W0801 05:53:52.314886 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    Instructions for updating:
[worker-0]:    INFO:tensorflow:Start training at 0
[worker-1]:    W0801 05:53:52.315736 281473656060608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    I0801 05:53:52.313986 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    Instructions for updating:
[worker-2]:    INFO:tensorflow:Start training at 0
[worker-2]:    I0801 05:53:52.315163 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    INFO:tensorflow:Start training at 0
[worker-1]:    I0801 05:53:52.315945 281473656060608 failure_handler_test.py:197] Start training at 0
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.474638 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.488862 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.494473 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.489393 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.600728 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.600828 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.607844 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.621567 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.707532 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.707710 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.716954 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.729718 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.797745 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.798659 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.798913 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.798998 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.861972 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.862568 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.862499 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbe20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb8cc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbae80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:52.914480 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbbe20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:52.914395 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb8cc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:52.914552 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbae80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.924626 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.924714 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.924826 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbaac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb9260> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb9800> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:52.976575 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdbaac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:52.976495 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb9260> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:52.976646 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb9800> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.986763 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.986859 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.986807 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.048716 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.047680 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.107987 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.107999 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.224185 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.233651 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.048079 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.108636 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.229669 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.860728 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb91c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:52.914111 281473656060608 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb91c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.922718 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb96c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:52.976246 281473656060608 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffffacdb96c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.985838 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.046243 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.107153 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.230964 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.337117 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.336611 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.347062 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.346002 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:sending sigterm
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:54.019349 281473864792768 failure_handler_test.py:302] sending sigterm
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.539825 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.540086 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.540840 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.540724 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.601698 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[  FAILED  ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker
[worker-1]:    I0801 05:53:53.600558 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 56.42s
I0801 05:54:47.151715 281473864792768 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 56.42s
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
======================================================================
ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest)
PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker
test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='checkpoint', strategy_option='MWMS_multi_worker')
[worker-0]:    I0801 05:53:53.600558 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test
    return test_method(self, **testcase_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated
    execute_test_method()
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator
    test_method(self, **kwargs)
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing
    os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM)
ProcessLookupError: [Errno 3] No such process
[worker-1]:    I0801 05:53:53.658691 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1

[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
----------------------------------------------------------------------
Ran 3 tests in 96.322s
[worker-3]:    I0801 05:53:53.658762 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1

[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
FAILED (errors=1)
[worker-0]:    I0801 05:53:53.659847 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.600765 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.716134 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.716164 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 0 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 0 finished
[worker-2]:    I0801 05:53:53.659030 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.716135 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.764746 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-1]:    I0801 05:53:53.765104 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 0 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.716360 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.764974 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-3]:    I0801 05:53:53.773908 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.773881 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 0 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.831032 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.765157 281473656060608 failure_handler_test.py:195] epoch 0 finished
[worker-0]:    I0801 05:53:53.774069 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.831012 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.886585 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.773968 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.831012 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.886458 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.942474 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.831212 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.886409 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.942450 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.998654 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.887274 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.942454 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.998636 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.054808 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.942685 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.998993 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.054783 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.110931 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.998735 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.054804 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.110913 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.170600 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.054883 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.110911 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.170574 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.229622 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.112151 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.170581 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.229612 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.170787 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.284284 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.229613 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.284237 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.229794 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.341043 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.284242 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.341026 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.284440 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.398099 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.341032 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.397946 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.341182 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.454672 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.397625 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.454779 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.399654 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.508592 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.454727 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.508581 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.454289 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.562495 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.508949 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.562503 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.508089 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.608288 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-0]:    I0801 05:53:54.562486 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 1 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 1 finished
[worker-1]:    I0801 05:53:54.608585 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-2]:    I0801 05:53:54.561874 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.616248 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.608440 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 1 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.616349 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.608618 281473656060608 failure_handler_test.py:195] epoch 1 finished
[worker-3]:    I0801 05:53:54.669039 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.616237 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.669066 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.616136 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.721967 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.669028 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.721991 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.668796 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.775254 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.721968 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.775247 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.721694 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.831122 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.776328 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.831108 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.776593 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.886341 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.830552 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.886372 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.939441 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.831220 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.939560 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.885865 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.886883 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:54.992021 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:54.992173 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.939084 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.939289 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.044824 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.044919 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:54.991674 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.991888 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.097248 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.097282 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.044495 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.044709 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.150066 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.150192 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.096926 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.097067 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.202177 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.202305 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.149708 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.149851 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.256461 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.257178 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.202009 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.202213 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.310128 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.310643 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.258162 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.256713 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.362446 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.310333 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.363023 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 2 finished
[worker-2]:    I0801 05:53:55.310657 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 2 finished
[worker-3]:    I0801 05:53:55.407371 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.362524 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.407697 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.362931 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.414602 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 2 finished
[worker-2]:    INFO:tensorflow:epoch 2 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.415160 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.407551 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    I0801 05:53:55.407723 281473656060608 failure_handler_test.py:195] epoch 2 finished
[worker-3]:    I0801 05:53:55.466860 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.468442 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.414760 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.415670 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.521852 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.522879 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.467040 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.467436 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.574172 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.574639 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.522486 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.522742 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.625776 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.626656 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.574274 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.574629 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.679209 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.679651 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.626364 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.627833 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.730897 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.730984 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.679362 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.781749 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.679744 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.730855 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.782078 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.832999 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.731255 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.781932 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.832882 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.884093 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.782071 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.832823 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.884377 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.935414 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.833448 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.884241 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:55.986444 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.884366 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.935334 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.935247 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.935909 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.037710 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:55.986725 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:55.986541 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:55.986726 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.088562 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.037626 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.037516 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.038151 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.141289 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.088847 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 3 finished
[worker-0]:    I0801 05:53:56.088729 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.141178 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.088878 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.185470 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-1]:    INFO:tensorflow:epoch 3 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.185756 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-0]:    I0801 05:53:56.141103 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.141706 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.192649 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 3 finished
[worker-2]:    INFO:tensorflow:epoch 3 finished
[worker-1]:    I0801 05:53:56.192978 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.185787 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-0]:    I0801 05:53:56.185614 281473656060608 failure_handler_test.py:195] epoch 3 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.244023 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.243912 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.193098 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.192818 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.294955 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.295181 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.244415 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.243805 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.346159 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.346076 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.295082 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.295227 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.397094 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.397406 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.345939 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.346543 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.448434 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.448300 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.397181 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.397392 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.499217 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.499491 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.448177 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.448811 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.550443 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.550672 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.499346 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.499508 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.601721 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.601548 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.550447 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.550992 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.652606 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.652666 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.601658 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.601769 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.704055 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.703709 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.652549 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.653131 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.756877 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.756833 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.703783 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.704040 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.808206 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.808019 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.756803 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.757360 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.859318 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.859187 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.808252 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.808256 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.910539 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.910187 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.859100 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 4 finished
[worker-2]:    I0801 05:53:56.859757 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 4 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:56.954625 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:56.954356 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-0]:    I0801 05:53:56.910430 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.910540 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 4 finished
[worker-1]:    I0801 05:53:56.961962 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 4 finished
[worker-3]:    I0801 05:53:56.962086 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.954513 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.954664 281473656060608 failure_handler_test.py:195] epoch 4 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.013204 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:56.961875 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.013035 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:56.962530 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.064055 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.013194 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.064176 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.013230 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.115327 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.063954 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.115028 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.064595 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.166226 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.115171 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.166452 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.115312 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.217535 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.166236 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.217303 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.166825 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.268362 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.217489 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.268533 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.217526 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.319711 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.319316 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.268940 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.268326 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.372382 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.372617 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.319689 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.319517 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.423707 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.423462 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.372974 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.372413 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.474569 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.474689 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.423697 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.423555 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.525758 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.475094 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.474472 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.525549 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.576738 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.525634 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.576653 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.627924 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.678820 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 5 finished
[worker-0]:    I0801 05:53:57.723235 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.730592 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.781545 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.832866 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.883923 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.935151 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.525726 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:57.987855 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.039140 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.090118 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.141423 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.192334 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.243516 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.294521 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.345648 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.396852 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.448022 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 6 finished
[worker-0]:    I0801 05:53:58.492286 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.499579 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.550965 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.603435 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.654765 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.705633 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.756903 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.807996 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.859042 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.910147 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:58.961387 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:59.012402 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:59.063641 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:59.114608 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:59.165736 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:59.218400 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 7 finished
[worker-0]:    I0801 05:53:59.262800 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-0]:    INFO:tensorflow:Training finished.
[worker-0]:    I0801 05:53:59.263821 281473656060608 failure_handler_test.py:245] Training finished.
[worker-1]:    I0801 05:53:57.627966 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.678925 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 5 finished
[worker-1]:    I0801 05:53:57.723342 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.730725 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.781633 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.833024 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.883968 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.935360 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:57.987905 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.039314 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.090246 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.141533 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.192450 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.243669 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.294608 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.577214 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.345828 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.576867 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.627706 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.678994 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 5 finished
[worker-3]:    I0801 05:53:57.723088 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.730407 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.781850 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.832737 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.884103 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.935030 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:57.987995 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.039063 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.090335 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.141299 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.192538 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.243377 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.294676 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.345557 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.397010 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.447834 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 6 finished
[worker-3]:    I0801 05:53:58.492137 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.499833 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.550867 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.603559 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.654669 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.705853 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.756748 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.808104 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.858924 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.910341 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:58.961273 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:59.012605 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:59.063556 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:59.114805 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:59.165637 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:59.218656 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 7 finished
[worker-3]:    I0801 05:53:59.262668 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-3]:    INFO:tensorflow:Training finished.
[worker-3]:    I0801 05:53:59.263409 281473656060608 failure_handler_test.py:245] Training finished.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.627978 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.396848 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.679409 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.448159 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 5 finished
[worker-1]:    INFO:tensorflow:epoch 6 finished
[worker-2]:    I0801 05:53:57.723381 281473656060608 failure_handler_test.py:195] epoch 5 finished
[worker-1]:    I0801 05:53:58.492400 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.730755 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.499686 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.782196 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.551101 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.833006 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.603536 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.884538 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.654929 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.935304 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.705715 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:57.988412 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.757067 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.039306 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.808012 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.090688 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.859122 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.141598 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.910138 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.192893 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:58.961473 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.243679 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:59.012626 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.295065 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:59.063821 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.345841 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:59.114723 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.397300 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:59.165955 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.448148 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:59.218454 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 6 finished
[worker-1]:    INFO:tensorflow:epoch 7 finished
[worker-2]:    I0801 05:53:58.492445 281473656060608 failure_handler_test.py:195] epoch 6 finished
[worker-1]:    I0801 05:53:59.262936 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Training finished.
[worker-2]:    I0801 05:53:58.500367 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:59.263968 281473656060608 failure_handler_test.py:245] Training finished.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.551135 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.604045 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.654925 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.706253 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.757058 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.808469 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.859164 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.910636 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:58.961443 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:59.013045 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:59.063754 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:59.115183 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:59.165937 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:59.218987 281473656060608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 7 finished
[worker-2]:    I0801 05:53:59.262968 281473656060608 failure_handler_test.py:195] epoch 7 finished
[worker-2]:    INFO:tensorflow:Training finished.
[worker-2]:    I0801 05:53:59.264074 281473656060608 failure_handler_test.py:245] Training finished.
================================================================================
==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 1 of 8):
2023-08-01 05:53:02.252722: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Running tests under Python 3.11.1: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3
[ RUN      ] PreemptionCheckpointTest.test_error_propagation
INFO:tensorflow:Using local port 43813
I0801 05:53:10.835267 281473568242368 test_util.py:3813] Using local port 43813
INFO:tensorflow:Using local port 34561
I0801 05:53:10.836013 281473568242368 test_util.py:3813] Using local port 34561
INFO:tensorflow:Using local port 37103
I0801 05:53:10.836442 281473568242368 test_util.py:3813] Using local port 37103
INFO:tensorflow:Using local port 42475
I0801 05:53:10.836826 281473568242368 test_util.py:3813] Using local port 42475
2023-08-01 05:53:12.632423: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-01 05:53:12.723184: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
INFO:tensorflow:Cluster starting.
I0801 05:53:15.749744 281473568242368 failure_handler_test.py:387] Cluster starting.
[worker-0]:    I0801 05:53:15.875516 281473270840000 multi_process_runner.py:840] Subprocess with PID 2912731 (worker, 0) is now being started.
[worker-0]:    I0801 05:53:15.875921 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43813", "localhost:34561", "localhost:37103", "localhost:42475"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-1]:    I0801 05:53:16.013258 281473270840000 multi_process_runner.py:840] Subprocess with PID 2912792 (worker, 1) is now being started.
[worker-3]:    I0801 05:53:16.032404 281473270840000 multi_process_runner.py:840] Subprocess with PID 2913239 (worker, 3) is now being started.
[worker-1]:    I0801 05:53:16.013659 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43813", "localhost:34561", "localhost:37103", "localhost:42475"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-0]:    2023-08-01 05:53:16.056949: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:43813
[worker-3]:    I0801 05:53:16.032821 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43813", "localhost:34561", "localhost:37103", "localhost:42475"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-2]:    I0801 05:53:16.065066 281473270840000 multi_process_runner.py:840] Subprocess with PID 2912992 (worker, 2) is now being started.
[worker-0]:    2023-08-01 05:53:16.068472: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 4658965107066568830
[worker-0]:    2023-08-01 05:53:16.068851: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    I0801 05:53:16.065484 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43813", "localhost:34561", "localhost:37103", "localhost:42475"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-2]:    2023-08-01 05:53:16.109596: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37103
[worker-1]:    2023-08-01 05:53:16.137653: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34561
[worker-0]:    2023-08-01 05:53:16.142972: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 8210185553941129364
[worker-1]:    2023-08-01 05:53:16.143293: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:16.163227: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 17959197533926704401
[worker-2]:    2023-08-01 05:53:16.164081: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    2023-08-01 05:53:16.176878: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42475
[worker-0]:    2023-08-01 05:53:16.195998: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 16623716656483531063
[worker-3]:    2023-08-01 05:53:16.200809: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    I0801 05:53:16.202886 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-2]:    I0801 05:53:16.202957 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    I0801 05:53:16.203564 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:16.207517 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    I0801 05:53:16.271062 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:16.271573 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:16.271811 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-0]:    I0801 05:53:16.273295 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-2]:    I0801 05:53:16.273290 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    I0801 05:53:16.273795 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    I0801 05:53:16.273819 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:16.274031 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:16.274057 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    I0801 05:53:16.283703 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:16.284270 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:16.284515 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43813', 'localhost:34561', 'localhost:37103', 'localhost:42475']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    I0801 05:53:16.348332 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:16.349081 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    I0801 05:53:16.349334 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    W0801 05:53:16.349658 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    I0801 05:53:16.350446 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    Instructions for updating:
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    I0801 05:53:16.351657 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:Start training at 0
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    I0801 05:53:16.349844 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-2]:    I0801 05:53:16.351920 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    W0801 05:53:16.352242 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    INFO:tensorflow:Start training at 0
[worker-2]:    I0801 05:53:16.352430 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    I0801 05:53:16.371657 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    I0801 05:53:16.375149 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-3]:    I0801 05:53:16.381325 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-3]:    I0801 05:53:16.381671 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    W0801 05:53:16.382002 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start training at 0
[worker-3]:    I0801 05:53:16.382189 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-1]:    I0801 05:53:16.386754 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-1]:    I0801 05:53:16.387085 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    W0801 05:53:16.387407 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    INFO:tensorflow:Start training at 0
[worker-1]:    I0801 05:53:16.387595 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-2]:    INFO:tensorflow:Error reported to Coordinator: in user code:
[worker-2]:    
[worker-0]:    2023-08-01 05:53:16.578068: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:990] /job:worker/replica:0/task:2 has been set to ERROR in coordination service: RESOURCE_EXHAUSTED: in user code:
[worker-0]:    
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-0]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:    Traceback (most recent call last):
[worker-2]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception
[worker-2]:        yield
[worker-2]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run
[worker-2]:        self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
[worker-2]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[worker-2]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper
[worker-2]:        raise e.ag_error_metadata.to_exception(e)
[worker-2]:    tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code:
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:    
[worker-2]:    I0801 05:53:16.571570 281448015327712 coordinator.py:213] Error reported to Coordinator: in user code:
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:    Traceback (most recent call last):
[worker-2]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception
[worker-2]:        yield
[worker-2]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run
[worker-2]:        self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
[worker-2]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[worker-0]:            raise errors_impl.ResourceExhaustedError(
[worker-0]:    
[worker-0]:        ResourceExhaustedError: Running out of resources
[worker-0]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-2]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper
[worker-2]:        raise e.ag_error_metadata.to_exception(e)
[worker-2]:    tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code:
[worker-1]:    2023-08-01 05:53:16.588196: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code:
[worker-1]:    
[worker-3]:    2023-08-01 05:53:16.589343: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code:
[worker-3]:    
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-3]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-1]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-3]:            raise errors_impl.ResourceExhaustedError(
[worker-1]:            raise errors_impl.ResourceExhaustedError(
[worker-3]:    
[worker-1]:    
[worker-3]:        ResourceExhaustedError: Running out of resources
[worker-1]:        ResourceExhaustedError: Running out of resources
[worker-3]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-1]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-3]:    2023-08-01 05:53:16.589394: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code:
[worker-1]:    2023-08-01 05:53:16.588268: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code:
[worker-3]:    
[worker-1]:    
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-3]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-1]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-3]:            raise errors_impl.ResourceExhaustedError(
[worker-1]:            raise errors_impl.ResourceExhaustedError(
[worker-3]:    
[worker-1]:    
[worker-3]:        ResourceExhaustedError: Running out of resources
[worker-1]:        ResourceExhaustedError: Running out of resources
[worker-3]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:    
[worker-2]:    INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): in user code:
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-2]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:    
[worker-2]:    I0801 05:53:16.577146 281473270840000 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): in user code:
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-2]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:    
[worker-1]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-2]:    2023-08-01 05:53:16.577436: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: in user code:
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:16.599276 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    2023-08-01 05:53:16.606290: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code:
[worker-0]:    
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-0]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-0]:            raise errors_impl.ResourceExhaustedError(
[worker-0]:    
[worker-0]:        ResourceExhaustedError: Running out of resources
[worker-0]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-0]:    2023-08-01 05:53:16.606356: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code:
[worker-0]:    
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-0]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-0]:            raise errors_impl.ResourceExhaustedError(
[worker-0]:    
[worker-0]:        ResourceExhaustedError: Running out of resources
[worker-0]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-2]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-2]:    2023-08-01 05:53:16.577497: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: in user code:
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-2]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:     [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02']
[worker-2]:    2023-08-01 05:53:16.577521: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:434] Reporting error to coordination service: RESOURCE_EXHAUSTED: in user code:
[worker-2]:    
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-2]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-2]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-2]:            raise errors_impl.ResourceExhaustedError(
[worker-2]:    
[worker-2]:        ResourceExhaustedError: Running out of resources
[worker-2]:    
[worker-0]:    INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error:
[worker-0]:    
[worker-0]:    Detected at node CollectiveReduceV2 defined at (most recent call last):
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in <module>
[worker-0]:    
[worker-0]:      File "<string>", line 1, in <module>
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 133, in _main
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn
[worker-0]:    
[worker-0]:    Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:
[worker-0]:    
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-0]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-0]:            raise errors_impl.ResourceExhaustedError(
[worker-0]:    
[worker-0]:        ResourceExhaustedError: Running out of resources
[worker-0]:    
[worker-0]:    The error could be from a previous operation. Restart your program to reset.
[worker-0]:    	 [[{{node CollectiveReduceV2}}]]
[worker-0]:    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[worker-0]:     [Op:__inference_train_step_40]
[worker-0]:    I0801 05:53:16.671541 281473270840000 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error:
[worker-0]:    
[worker-0]:    Detected at node CollectiveReduceV2 defined at (most recent call last):
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in <module>
[worker-0]:    
[worker-0]:      File "<string>", line 1, in <module>
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 133, in _main
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step
[worker-0]:    
[worker-0]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn
[worker-0]:    
[worker-0]:    Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:
[worker-0]:    
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-0]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-0]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-0]:            raise errors_impl.ResourceExhaustedError(
[worker-0]:    
[worker-0]:        ResourceExhaustedError: Running out of resources
[worker-0]:    
[worker-0]:    The error could be from a previous operation. Restart your program to reset.
[worker-0]:    	 [[{{node CollectiveReduceV2}}]]
[worker-0]:    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[worker-0]:     [Op:__inference_train_step_40]
[worker-0]:    INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state.
[worker-0]:    I0801 05:53:16.671949 281473270840000 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:16.699941 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:16.715933 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error:
[worker-1]:    
[worker-1]:    Detected at node CollectiveReduceV2 defined at (most recent call last):
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in <module>
[worker-1]:    
[worker-1]:      File "<string>", line 1, in <module>
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 133, in _main
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn
[worker-1]:    
[worker-1]:    Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:
[worker-1]:    
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-1]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-1]:            raise errors_impl.ResourceExhaustedError(
[worker-1]:    
[worker-1]:        ResourceExhaustedError: Running out of resources
[worker-1]:    
[worker-1]:    The error could be from a previous operation. Restart your program to reset.
[worker-1]:    	 [[{{node CollectiveReduceV2}}]]
[worker-1]:    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[worker-1]:     [Op:__inference_train_step_38]
[worker-1]:    I0801 05:53:16.797465 281473270840000 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error:
[worker-1]:    
[worker-1]:    Detected at node CollectiveReduceV2 defined at (most recent call last):
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in <module>
[worker-1]:    
[worker-1]:      File "<string>", line 1, in <module>
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 133, in _main
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step
[worker-1]:    
[worker-1]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn
[worker-1]:    
[worker-1]:    Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:
[worker-1]:    
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-1]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-1]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-1]:            raise errors_impl.ResourceExhaustedError(
[worker-1]:    
[worker-1]:        ResourceExhaustedError: Running out of resources
[worker-1]:    
[worker-1]:    The error could be from a previous operation. Restart your program to reset.
[worker-1]:    	 [[{{node CollectiveReduceV2}}]]
[worker-1]:    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[worker-1]:     [Op:__inference_train_step_38]
[worker-1]:    INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state.
[worker-1]:    I0801 05:53:16.797891 281473270840000 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state.
[worker-3]:    INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error:
[worker-3]:    
[worker-3]:    Detected at node CollectiveReduceV2 defined at (most recent call last):
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in <module>
[worker-3]:    
[worker-3]:      File "<string>", line 1, in <module>
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 133, in _main
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn
[worker-3]:    
[worker-3]:    Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:
[worker-3]:    
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-3]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-3]:            raise errors_impl.ResourceExhaustedError(
[worker-3]:    
[worker-3]:        ResourceExhaustedError: Running out of resources
[worker-3]:    
[worker-3]:    The error could be from a previous operation. Restart your program to reset.
[worker-3]:    	 [[{{node CollectiveReduceV2}}]]
[worker-3]:    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[worker-3]:     [Op:__inference_train_step_38]
[worker-3]:    Exception in thread PeerTerminationWatcher-3:
[worker-3]:    Traceback (most recent call last):
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
[worker-3]:        self.run()
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run
[worker-3]:        self._target(*self._args, **self._kwargs)
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1229, in _watch_step_to_save_key
[worker-3]:        step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY)
[worker-3]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 806, in get_config_key_value
[worker-3]:        pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key,
[worker-3]:    tensorflow.python.framework.errors_impl.UnavailableError: Socket closed
[worker-3]:    Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/GetKeyValue:
[worker-3]:    :{"created":"@1690869196.838868408","description":"Error received from peer ipv6:[::1]:43813","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}
[worker-3]:    I0801 05:53:16.840831 281473270840000 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error:
[worker-3]:    
[worker-3]:    Detected at node CollectiveReduceV2 defined at (most recent call last):
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in <module>
[worker-3]:    
[worker-3]:      File "<string>", line 1, in <module>
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 133, in _main
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step
[worker-3]:    
[worker-3]:      File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn
[worker-3]:    
[worker-3]:    Collective ops is aborted by: Error reported from /job:worker/task:2: in user code:
[worker-3]:    
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn  *
[worker-3]:            return call_for_each_replica(strategy, fn.python_function, args, kwargs)
[worker-3]:        File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step  *
[worker-3]:            raise errors_impl.ResourceExhaustedError(
[worker-3]:    
[worker-3]:        ResourceExhaustedError: Running out of resources
[worker-3]:    
[worker-3]:    The error could be from a previous operation. Restart your program to reset.
[worker-3]:    	 [[{{node CollectiveReduceV2}}]]
[worker-3]:    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[worker-3]:     [Op:__inference_train_step_38]
[worker-3]:    INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state.
[worker-3]:    I0801 05:53:16.864715 281473270840000 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state.
I0801 05:53:17.791105 281473568242368 multi_process_runner.py:646] worker-0 exit code: 0
I0801 05:53:17.791461 281473568242368 multi_process_runner.py:646] worker-1 exit code: 0
I0801 05:53:17.791644 281473568242368 multi_process_runner.py:646] worker-2 exit code: 0
I0801 05:53:17.791826 281473568242368 multi_process_runner.py:646] worker-3 exit code: 0
I0801 05:53:17.796434 281473568242368 multi_process_runner.py:662] Joining log reading threads.
I0801 05:53:17.796751 281473568242368 multi_process_runner.py:665] Joined log reading threads.
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_error_propagation): 7.18s
I0801 05:53:18.008261 281473568242368 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_error_propagation): 7.18s
[       OK ] PreemptionCheckpointTest.test_error_propagation
[ RUN      ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice
INFO:tensorflow:Start watcher for local signal.
I0801 05:53:18.145083 281473568242368 failure_handling.py:674] Start watcher for local signal.
INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
I0801 05:53:18.145511 281473568242368 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
Instructions for updating:
Track steps using a tf.Variable saved in checkpoint instead.
W0801 05:53:18.145846 281473568242368 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
Instructions for updating:
Track steps using a tf.Variable saved in checkpoint instead.
INFO:tensorflow:Start training at 0
I0801 05:53:18.146042 281473568242368 failure_handler_test.py:197] Start training at 0
WARNING:tensorflow:5 out of the last 5 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xfffefd69e840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
W0801 05:53:18.345129 281473568242368 polymorphic_function.py:156] 5 out of the last 5 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xfffefd69e840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xfffefd69e840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
W0801 05:53:18.362447 281473568242368 polymorphic_function.py:156] 6 out of the last 6 calls to <function PreemptionCheckpointTest.worker_fn.<locals>.distributed_train_step.<locals>.train_step at 0xfffefd69e840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
INFO:tensorflow:epoch 0 finished
I0801 05:53:18.494979 281473568242368 failure_handler_test.py:195] epoch 0 finished
INFO:tensorflow:epoch 1 finished
I0801 05:53:18.728240 281473568242368 failure_handler_test.py:195] epoch 1 finished
INFO:tensorflow:epoch 2 finished
I0801 05:53:19.042783 281473568242368 failure_handler_test.py:195] epoch 2 finished
INFO:tensorflow:epoch 3 finished
I0801 05:53:19.274488 281473568242368 failure_handler_test.py:195] epoch 3 finished
INFO:tensorflow:epoch 4 finished
I0801 05:53:19.919892 281473568242368 failure_handler_test.py:195] epoch 4 finished
INFO:tensorflow:epoch 5 finished
I0801 05:53:20.174360 281473568242368 failure_handler_test.py:195] epoch 5 finished
INFO:tensorflow:epoch 6 finished
I0801 05:53:20.596939 281473568242368 failure_handler_test.py:195] epoch 6 finished
INFO:tensorflow:epoch 7 finished
I0801 05:53:20.822988 281473568242368 failure_handler_test.py:195] epoch 7 finished
INFO:tensorflow:Training finished.
I0801 05:53:20.823593 281473568242368 failure_handler_test.py:245] Training finished.
INFO:tensorflow:sending sigterm
I0801 05:53:21.026329 281470629114336 failure_handler_test.py:467] sending sigterm
INFO:tensorflow:Member single_worker has received termination notice.
I0801 05:53:21.186180 281473568242368 failure_handling.py:701] Member single_worker has received termination notice.
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 3.18s
I0801 05:53:21.186877 281473568242368 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 3.18s
[       OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice
[ RUN      ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker
INFO:tensorflow:Using local port 45025
I0801 05:53:21.189416 281473568242368 test_util.py:3813] Using local port 45025
INFO:tensorflow:Using local port 37273
I0801 05:53:21.189815 281473568242368 test_util.py:3813] Using local port 37273
INFO:tensorflow:Using local port 43749
I0801 05:53:21.190175 281473568242368 test_util.py:3813] Using local port 43749
INFO:tensorflow:Using local port 35881
I0801 05:53:21.190541 281473568242368 test_util.py:3813] Using local port 35881
INFO:tensorflow:Cluster starting.
I0801 05:53:21.240074 281473568242368 failure_handler_test.py:297] Cluster starting.
[worker-0]:    I0801 05:53:21.319179 281473270840000 multi_process_runner.py:840] Subprocess with PID 2929689 (worker, 0) is now being started.
[worker-0]:    I0801 05:53:21.319600 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45025", "localhost:37273", "localhost:43749", "localhost:35881"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-0]:    2023-08-01 05:53:21.400675: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:45025
[worker-1]:    I0801 05:53:21.424468 281473270840000 multi_process_runner.py:840] Subprocess with PID 2929699 (worker, 1) is now being started.
[worker-1]:    I0801 05:53:21.424900 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45025", "localhost:37273", "localhost:43749", "localhost:35881"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-2]:    I0801 05:53:21.432585 281473270840000 multi_process_runner.py:840] Subprocess with PID 2929713 (worker, 2) is now being started.
[worker-0]:    2023-08-01 05:53:21.443657: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 773203135660262267
[worker-0]:    2023-08-01 05:53:21.443893: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    I0801 05:53:21.432981 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45025", "localhost:37273", "localhost:43749", "localhost:35881"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:21.484330 281473270840000 multi_process_runner.py:840] Subprocess with PID 2929739 (worker, 3) is now being started.
[worker-3]:    I0801 05:53:21.484755 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:45025", "localhost:37273", "localhost:43749", "localhost:35881"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-1]:    2023-08-01 05:53:21.502063: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37273
[worker-0]:    2023-08-01 05:53:21.520308: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2078303125949512871
[worker-1]:    2023-08-01 05:53:21.520513: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    2023-08-01 05:53:21.532260: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35881
[worker-0]:    2023-08-01 05:53:21.556182: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 4875736947353070314
[worker-3]:    2023-08-01 05:53:21.556419: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    2023-08-01 05:53:21.595736: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:43749
[worker-0]:    2023-08-01 05:53:21.607074: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 4454901966366615082
[worker-2]:    2023-08-01 05:53:21.608072: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-3]:    I0801 05:53:21.618662 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:21.619452 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    I0801 05:53:21.627045 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    I0801 05:53:21.620747 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    I0801 05:53:21.690835 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:21.691553 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:21.691801 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    I0801 05:53:21.694949 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:21.695501 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:21.695737 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    I0801 05:53:21.697619 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    I0801 05:53:21.698117 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:21.698349 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-0]:    I0801 05:53:21.712331 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    I0801 05:53:21.712883 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:21.713121 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:45025', 'localhost:37273', 'localhost:43749', 'localhost:35881']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    I0801 05:53:21.807104 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-1]:    I0801 05:53:21.808937 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-1]:    I0801 05:53:21.809231 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    W0801 05:53:21.809557 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    INFO:tensorflow:Start training at 0
[worker-1]:    I0801 05:53:21.809744 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-2]:    I0801 05:53:21.815299 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    I0801 05:53:21.830500 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:21.831210 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    I0801 05:53:21.831452 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    W0801 05:53:21.831768 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    I0801 05:53:21.829562 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    Instructions for updating:
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    I0801 05:53:21.830341 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    INFO:tensorflow:Start training at 0
[worker-3]:    I0801 05:53:21.830626 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    I0801 05:53:21.831952 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    W0801 05:53:21.830938 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start training at 0
[worker-3]:    I0801 05:53:21.831120 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-2]:    I0801 05:53:21.838832 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    I0801 05:53:21.839168 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    W0801 05:53:21.839478 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    INFO:tensorflow:Start training at 0
[worker-2]:    I0801 05:53:21.839663 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:21.977357 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.094734 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.096646 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.181664 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.294447 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.303909 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.300742 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.331385 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.414266 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.440597 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.431439 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.444251 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.541709 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.529841 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.547760 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.547929 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.634515 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.645846 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.641427 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.644389 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d6c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:22.722858 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d6c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5ef20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:22.723777 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5ef20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.736393 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.741069 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d3a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:22.746965 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d3a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e9e200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:22.749094 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e9e200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.761848 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.757698 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5e980> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e9f420> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:22.820292 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e9f420> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:22.819853 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5e980> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5f7e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:22.826463 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5f7e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5df80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:22.826045 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5df80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.830433 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.840721 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.835829 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.860548 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:22.945159 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:22.952103 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:22.985576 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:22.987075 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.051647 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.055988 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.070596 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.100356 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.213363 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.217569 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.230459 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.235162 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.358382 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.358828 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.363477 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.369684 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:sending sigterm
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:24.363124 281473568242368 failure_handler_test.py:302] sending sigterm
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[  FAILED  ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker
[worker-0]:    I0801 05:53:23.615973 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.621567 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 18.16s
[worker-3]:    I0801 05:53:23.631087 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:39.351785 281473568242368 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 18.16s
[ RUN      ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.620748 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.687381 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.688160 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:Using local port 35277
[worker-2]:    I0801 05:53:23.688159 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:39.355464 281473568242368 test_util.py:3813] Using local port 35277
INFO:tensorflow:Using local port 35081
[worker-1]:    I0801 05:53:23.687527 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:39.356063 281473568242368 test_util.py:3813] Using local port 35081
INFO:tensorflow:Using local port 39459
I0801 05:53:39.357899 281473568242368 test_util.py:3813] Using local port 39459
INFO:tensorflow:Using local port 36055
I0801 05:53:39.358306 281473568242368 test_util.py:3813] Using local port 36055
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.745040 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.745264 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.803457 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 0 finished
[worker-2]:    I0801 05:53:23.745022 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.802591 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.745021 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.853052 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 0 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.852668 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-2]:    I0801 05:53:23.803061 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 0 finished
[worker-1]:    I0801 05:53:23.804028 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.861901 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.853342 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-3]:    I0801 05:53:23.862564 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 0 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.862978 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.921849 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.853191 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-3]:    I0801 05:53:23.921930 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.862887 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:23.986250 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:23.985377 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.921402 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.043904 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.921975 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.043700 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:23.985777 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.105662 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:23.986040 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.106558 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.164859 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.043730 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.043807 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.165320 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.106910 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.224113 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.106780 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.224686 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.165465 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.281404 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.168470 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.341157 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.224143 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.279698 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.224317 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.397612 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.281719 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.338855 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.281886 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.454950 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.341526 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.511032 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.396386 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.340911 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.397969 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.567733 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.453068 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.398352 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.624202 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.454749 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.510003 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.679953 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    I0801 05:53:24.566774 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.454625 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.511904 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.727037 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.623247 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.568040 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.512034 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.736015 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.679520 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.622911 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.568027 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.790999 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 1 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.623825 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.727154 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-2]:    I0801 05:53:24.679698 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.845567 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 1 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.734940 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.680189 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.727447 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-1]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.901239 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.790043 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.727487 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:24.957091 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.735207 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.844634 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.736404 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.790234 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.900060 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.012519 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.844936 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:24.956345 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.066862 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.791278 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:Cluster starting.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:39.423092 281473568242368 failure_handler_test.py:297] Cluster starting.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.012340 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.900186 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.119730 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.845920 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.066796 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:24.956928 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.902072 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.172600 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.119695 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.013531 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:24.956493 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.172587 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.067257 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.225477 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.011912 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.225425 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.066422 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.278578 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.120134 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.278579 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.119335 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.332756 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.332727 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.172966 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.172297 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.385879 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.386081 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.225925 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.225045 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.438755 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.278948 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.438282 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.278118 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.333170 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.492521 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.491554 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.332343 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 2 finished
[worker-2]:    I0801 05:53:25.386188 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 2 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.538565 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    I0801 05:53:25.438726 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.386106 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.538393 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.547089 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.492491 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.439709 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 2 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.598955 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.546796 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.538732 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-1]:    I0801 05:53:25.493537 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.653890 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.705820 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.546583 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 2 finished
[worker-0]:    I0801 05:53:25.599258 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.757997 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.538681 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.653104 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.813182 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.546334 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.599577 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.866951 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.599632 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.653168 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.706093 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.652902 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.920152 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.757954 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.705960 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.706254 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:25.972973 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.759068 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.812901 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.813805 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.758053 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.026404 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.866814 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.866702 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.814133 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:39.490146 281473270840000 multi_process_runner.py:840] Subprocess with PID 2935988 (worker, 0) is now being started.
[worker-0]:    I0801 05:53:25.920027 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.919734 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.867285 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:25.972826 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:25.973827 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.920425 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:39.490615 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:26.080724 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.026332 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.026241 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.134442 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.080561 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.080021 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:25.973187 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.133889 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.187956 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.026579 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.134399 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.187216 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.187485 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.240679 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.081292 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.240631 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.134510 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.294787 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.240656 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 3 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.188072 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.294626 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.340336 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 3 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.240735 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.340616 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-3]:    I0801 05:53:26.348320 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.294758 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.348049 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.295129 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 3 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 3 finished
[worker-0]:    I0801 05:53:26.340465 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-1]:    I0801 05:53:26.400472 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.340703 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.400677 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.453001 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.348310 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.348643 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.506770 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.400627 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.400928 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:39.521994 281473270840000 multi_process_runner.py:840] Subprocess with PID 2936497 (worker, 1) is now being started.
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.453198 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.559482 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.453393 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    2023-08-01 05:53:39.527281: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35277
[worker-0]:    I0801 05:53:26.453171 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.507019 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.507296 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:39.522400 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-1]:    I0801 05:53:26.611823 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.559439 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.507014 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.559801 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.663927 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.611888 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.559570 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.716541 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.612130 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.611868 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.664012 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.768929 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.664000 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.664252 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    2023-08-01 05:53:39.540480: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 5370897048613531457
[worker-3]:    I0801 05:53:26.716539 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.820617 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    2023-08-01 05:53:39.540712: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.716769 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.768759 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.873531 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.716528 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.820543 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.927343 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.768808 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.769059 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:26.979177 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.820589 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.873107 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.030896 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.873292 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:26.927178 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.082546 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.927321 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 4 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.127187 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-3]:    I0801 05:53:26.978965 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:26.979587 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.820929 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.134613 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.030748 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.030774 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.186102 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:26.873408 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.082987 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.082415 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 4 finished
[worker-3]:    INFO:tensorflow:epoch 4 finished
[worker-2]:    I0801 05:53:26.927586 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.237982 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.127039 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-3]:    I0801 05:53:27.126883 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.289203 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    2023-08-01 05:53:39.559696: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35081
[worker-2]:    I0801 05:53:26.979416 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.134407 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:39.560435 281473270840000 multi_process_runner.py:840] Subprocess with PID 2937393 (worker, 2) is now being started.
[worker-1]:    I0801 05:53:27.341067 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.134323 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.186403 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.031107 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.392323 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.185865 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.237596 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    2023-08-01 05:53:39.563555: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 15139815497486531059
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.082775 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.444339 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:39.560910 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-3]:    I0801 05:53:27.237461 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 4 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    2023-08-01 05:53:39.563989: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.127273 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    I0801 05:53:27.497118 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.289591 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.288912 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.550530 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.134718 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.340546 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.340420 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.601655 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.186236 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.392682 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.392144 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.653226 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.237800 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.443831 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.443709 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.704400 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.289305 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.496890 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.756013 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.550269 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.497260 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.340795 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.601382 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.807112 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.550384 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.652989 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.392490 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.601506 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.860321 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.704086 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 5 finished
[worker-2]:    I0801 05:53:27.444091 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.653053 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.907010 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-2]:    I0801 05:53:27.497138 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.755568 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.704301 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.550586 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.914401 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.806809 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.755665 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.601793 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:27.966303 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.859115 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.806955 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 5 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.653237 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.017935 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.906725 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-0]:    I0801 05:53:27.861626 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 5 finished
[worker-1]:    I0801 05:53:28.069695 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.704508 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.913915 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.906928 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.120887 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.755863 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:27.965573 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.914291 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.173964 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.807153 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.017656 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:27.966269 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.225406 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.860404 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 5 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.069293 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.907066 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-0]:    I0801 05:53:28.018002 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.276929 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.120561 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.914464 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.328328 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.069595 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:27.965842 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.120881 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.379536 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.173662 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.017972 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.225235 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.431190 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.173882 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:39.592251 281473270840000 multi_process_runner.py:840] Subprocess with PID 2937681 (worker, 3) is now being started.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.482558 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.069630 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.225313 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.120933 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.276856 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.533605 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    2023-08-01 05:53:39.595968: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39459
[worker-2]:    I0801 05:53:28.173974 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.328287 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:39.592666 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-1]:    I0801 05:53:28.585065 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.379455 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.225555 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.636039 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.431211 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.277042 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 6 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.328382 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.482431 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.680541 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.379577 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.276735 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.533572 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.687881 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.430951 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.328088 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.739112 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.584927 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.482516 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.379173 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.792675 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.533632 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.636050 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.430542 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 6 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.844617 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.482158 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.680431 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-2]:    I0801 05:53:28.585059 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.895883 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.533258 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.687762 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.636096 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.946764 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 6 finished
[worker-0]:    I0801 05:53:28.739039 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.584639 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.680588 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.792656 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.688014 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:28.998187 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.635778 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 6 finished
[worker-0]:    I0801 05:53:28.844606 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.739240 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.049634 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.680249 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.895721 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.792393 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.687451 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    2023-08-01 05:53:39.611824: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 194725660343581714
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    2023-08-01 05:53:39.612048: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.100749 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.844471 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.946801 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.151743 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.895794 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.738915 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.946840 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:28.998037 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.202818 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.792061 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:28.998114 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.049616 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.253663 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.844154 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.100630 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.304634 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.049638 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.151744 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.895455 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.355658 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.100768 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.202756 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:28.946481 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.253685 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.408875 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.151884 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 7 finished
[worker-3]:    I0801 05:53:28.997692 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.304477 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.453621 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.202816 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Training finished.
[worker-3]:    I0801 05:53:29.049340 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.355600 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:29.454694 281473270840000 failure_handler_test.py:245] Training finished.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.253713 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:29.100456 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.408849 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.304619 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 7 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:29.151477 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.453543 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-2]:    I0801 05:53:29.355733 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Training finished.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:29.202463 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:29.454550 281473270840000 failure_handler_test.py:245] Training finished.
[worker-2]:    I0801 05:53:29.408489 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 7 finished
[worker-3]:    I0801 05:53:29.253335 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.453680 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Training finished.
[worker-3]:    I0801 05:53:29.304301 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:29.454813 281473270840000 failure_handler_test.py:245] Training finished.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:29.355347 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:29.408179 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 7 finished
[worker-3]:    I0801 05:53:29.453343 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-3]:    INFO:tensorflow:Training finished.
[worker-3]:    I0801 05:53:29.453942 281473270840000 failure_handler_test.py:245] Training finished.
[worker-3]:    2023-08-01 05:53:39.664964: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:36055
[worker-0]:    2023-08-01 05:53:39.685251: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 12775317875617276427
[worker-3]:    2023-08-01 05:53:39.685727: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    I0801 05:53:39.688169 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    I0801 05:53:39.688269 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:39.688169 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    I0801 05:53:39.688457 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-2]:    I0801 05:53:39.755112 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-0]:    I0801 05:53:39.755120 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    I0801 05:53:39.755742 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    I0801 05:53:39.755747 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:39.755994 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:39.755985 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    I0801 05:53:39.761424 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:39.761971 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:39.762211 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    I0801 05:53:39.774183 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:39.774722 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:39.774963 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    I0801 05:53:39.823300 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    I0801 05:53:39.823748 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-0]:    I0801 05:53:39.824570 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    I0801 05:53:39.825880 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    I0801 05:53:39.825001 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    I0801 05:53:39.824935 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    I0801 05:53:39.825847 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-3]:    I0801 05:53:39.825422 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    Instructions for updating:
[worker-1]:    I0801 05:53:39.827077 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-3]:    Instructions for updating:
[worker-0]:    W0801 05:53:39.825292 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    I0801 05:53:39.827364 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    Instructions for updating:
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    I0801 05:53:39.829524 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    W0801 05:53:39.825826 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-0]:    INFO:tensorflow:Start training at 0
[worker-1]:    Instructions for updating:
[worker-3]:    Instructions for updating:
[worker-2]:    I0801 05:53:39.829861 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    I0801 05:53:39.825492 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    W0801 05:53:39.827726 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-3]:    INFO:tensorflow:Start training at 0
[worker-1]:    Instructions for updating:
[worker-3]:    I0801 05:53:39.826073 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    W0801 05:53:39.830197 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    INFO:tensorflow:Start training at 0
[worker-2]:    Instructions for updating:
[worker-1]:    I0801 05:53:39.827917 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    INFO:tensorflow:Start training at 0
[worker-2]:    I0801 05:53:39.830387 281473270840000 failure_handler_test.py:197] Start training at 0
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.010470 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.023183 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.028886 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:sending sigterm
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:41.550693 281473568242368 failure_handler_test.py:302] sending sigterm
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
INFO:tensorflow:sigterm sent
I0801 05:53:42.387366 281473568242368 failure_handler_test.py:306] sigterm sent
[worker-2]:    I0801 05:53:40.057630 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.137170 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.137451 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.198200 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.136693 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.198181 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.257454 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.197641 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.257815 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.317364 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.257323 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.316764 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e58360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d300> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:40.365873 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e58360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    I0801 05:53:40.316463 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    W0801 05:53:40.366026 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d300> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5c900> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.375354 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    W0801 05:53:40.366087 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5c900> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    I0801 05:53:40.375028 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5ad40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5f240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:40.424181 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5ad40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:40.424484 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5f240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.432977 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.433333 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.490710 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.490825 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.550805 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.550928 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.610321 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.670283 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.876659 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.933276 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:40.991227 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.374843 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.050517 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    INFO:tensorflow:epoch 0 finished
[worker-2]:    W0801 05:53:40.424562 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.432904 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.100068 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.491806 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.109680 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.170342 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.228523 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.552528 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.610679 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.671010 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.287997 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.876582 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.346543 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.406365 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.933199 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.464673 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:40.991215 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.523234 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.050364 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.583971 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 0 finished
[worker-3]:    I0801 05:53:40.136828 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.198094 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.256551 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.315592 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d300> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:40.365622 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5d300> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.375367 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5eac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:40.424081 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e5eac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.432989 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.490737 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.551651 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.610399 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.671211 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.876534 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.933313 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:40.992371 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.050630 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 0 finished
[worker-3]:    I0801 05:53:41.099941 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.610650 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.671743 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.875944 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.933692 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:40.991320 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.050616 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 0 finished
[worker-1]:    I0801 05:53:41.100248 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.110832 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.170649 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.228208 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.288344 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.346097 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.109636 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.407085 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.170295 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.464696 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.523663 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.228967 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.583432 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.287968 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.641928 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.350004 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.700651 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.405727 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.758923 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.466943 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.819530 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.523552 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.100312 281473270840000 failure_handler_test.py:195] epoch 0 finished
[worker-1]:    I0801 05:53:41.876711 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.935220 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.583330 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 1 finished
[worker-2]:    I0801 05:53:41.110893 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.984385 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.641728 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.170358 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:41.993462 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.700616 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.229759 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.053457 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.759493 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.130457 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.289119 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.819390 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.188732 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.346935 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.246860 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.876744 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.405776 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.315816 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.935055 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.464844 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.373417 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 1 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.984030 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-2]:    I0801 05:53:41.523375 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:41.993546 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.584029 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.051643 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged
[worker-2]:    I0801 05:53:41.641449 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.464017 281447755346400 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.701044 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.642149 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.113887 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.700541 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.759976 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.188484 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.819145 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.759620 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.245522 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.315963 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.876059 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.819344 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.373225 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.876678 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.461200 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.934867 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged
[worker-0]:    INFO:tensorflow:epoch 1 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:42.464613 281447285584352 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged
[worker-2]:    I0801 05:53:41.934905 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.984198 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-2]:    INFO:tensorflow:epoch 1 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.984483 281473270840000 failure_handler_test.py:195] epoch 1 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:41.993200 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:41.994193 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.053299 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.054141 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.114720 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.114721 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.189112 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.189222 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.246952 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.246822 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:42.477206 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.316794 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.316107 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.375097 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.374662 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Member 2 has received termination notice.
[worker-2]:    I0801 05:53:42.393558 281473270840000 failure_handling.py:710] Member 2 has received termination notice.
[worker-2]:    INFO:tensorflow:Termination caught in main thread on preempted worker
[worker-2]:    I0801 05:53:42.461530 281473270840000 failure_handling.py:1159] Termination caught in main thread on preempted worker
[worker-2]:    INFO:tensorflow:RUN_TO_CHECKPOINT set to 38
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:42.490742 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.463053 281473270840000 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 38
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged
[worker-2]:    I0801 05:53:42.464085 281447554019808 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 0 received
[worker-2]:    I0801 05:53:42.499301 281473270840000 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged
[worker-0]:    I0801 05:53:42.506570 281446958494176 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 1 received
[worker-2]:    I0801 05:53:42.500164 281473270840000 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 2 received
[worker-2]:    I0801 05:53:42.516908 281473270840000 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received
[worker-2]:    INFO:tensorflow:Sigterm acknowledgement from replica 3 received
[worker-2]:    I0801 05:53:42.548018 281473270840000 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:42.558173 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-3]:    I0801 05:53:42.613207 281473270840000 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-0]:    I0801 05:53:42.637118 281473270840000 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-1]:    I0801 05:53:42.657973 281473270840000 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-3]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/workertemp_3/fh_ckpt
[worker-3]:    I0801 05:53:42.684612 281473270840000 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/workertemp_3/fh_ckpt
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-2]:    I0801 05:53:42.678103 281473270840000 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint.
[worker-3]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-3]:    I0801 05:53:42.726967 281473270840000 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-3]:    I0801 05:53:42.727239 281473270840000 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-2]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/workertemp_2/fh_ckpt
[worker-2]:    I0801 05:53:42.763981 281473270840000 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/workertemp_2/fh_ckpt
[worker-2]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-2]:    I0801 05:53:42.770644 281473270840000 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-2]:    I0801 05:53:42.770947 281473270840000 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-0]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/fh_ckpt
[worker-0]:    I0801 05:53:42.786939 281473270840000 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/fh_ckpt
[worker-1]:    INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/workertemp_1/fh_ckpt
[worker-1]:    I0801 05:53:42.813700 281473270840000 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90z086mj1h/tmpo8c1vnj7/workertemp_1/fh_ckpt
[worker-1]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-1]:    I0801 05:53:42.816502 281473270840000 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-1]:    I0801 05:53:42.816741 281473270840000 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-0]:    INFO:tensorflow:Shut down watcher for peer's termination signal.
[worker-0]:    I0801 05:53:42.818040 281473270840000 failure_handling.py:771] Shut down watcher for peer's termination signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting.
[worker-0]:    I0801 05:53:42.818320 281473270840000 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting.
INFO:tensorflow:restarting workers
I0801 05:53:44.416282 281473568242368 failure_handler_test.py:309] restarting workers
INFO:tensorflow:workers restarted
I0801 05:53:44.586250 281473568242368 failure_handler_test.py:313] workers restarted
[worker-0]:    I0801 05:53:44.619218 281473270840000 multi_process_runner.py:840] Subprocess with PID 2944241 (worker, 0) is now being started.
[worker-0]:    I0801 05:53:44.619776 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}'
[worker-1]:    I0801 05:53:44.816575 281473270840000 multi_process_runner.py:840] Subprocess with PID 2944245 (worker, 1) is now being started.
[worker-1]:    I0801 05:53:44.817060 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}'
[worker-2]:    I0801 05:53:44.843195 281473270840000 multi_process_runner.py:840] Subprocess with PID 2944315 (worker, 2) is now being started.
[worker-2]:    I0801 05:53:44.843655 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}'
[worker-0]:    2023-08-01 05:53:44.899277: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35277
[worker-3]:    I0801 05:53:44.962028 281473270840000 multi_process_runner.py:840] Subprocess with PID 2944321 (worker, 3) is now being started.
[worker-0]:    2023-08-01 05:53:44.976504: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 3759113278166546568
[worker-3]:    I0801 05:53:44.962450 281473270840000 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:35277", "localhost:35081", "localhost:39459", "localhost:36055"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}'
[worker-0]:    2023-08-01 05:53:44.976915: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-2]:    2023-08-01 05:53:44.983238: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39459
[worker-1]:    2023-08-01 05:53:44.996929: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35081
[worker-0]:    2023-08-01 05:53:45.006579: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 8369451349747099188
[worker-2]:    2023-08-01 05:53:45.017217: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    2023-08-01 05:53:45.017584: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2581056849280483529
[worker-1]:    2023-08-01 05:53:45.039602: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-3]:    2023-08-01 05:53:45.330480: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:36055
[worker-0]:    2023-08-01 05:53:45.434576: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 14423953098360750258
[worker-3]:    2023-08-01 05:53:45.435246: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected.
[worker-0]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-0]:    I0801 05:53:45.437537 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-2]:    I0801 05:53:45.438293 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-1]:    I0801 05:53:45.437887 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1']
[worker-3]:    INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-3]:    I0801 05:53:45.457059 281473270840000 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1']
[worker-1]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    I0801 05:53:45.515829 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',)
[worker-2]:    INFO:tensorflow:Check health not enabled.
[worker-2]:    I0801 05:53:45.516486 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-2]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-2]:    I0801 05:53:45.516764 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:45.511817 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',)
[worker-1]:    INFO:tensorflow:Check health not enabled.
[worker-1]:    I0801 05:53:45.512385 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-1]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-1]:    I0801 05:53:45.512657 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-0]:    I0801 05:53:45.561339 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',)
[worker-0]:    INFO:tensorflow:Check health not enabled.
[worker-0]:    I0801 05:53:45.562068 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-0]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-0]:    I0801 05:53:45.562342 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:45.564620 281473270840000 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',)
[worker-3]:    INFO:tensorflow:Check health not enabled.
[worker-3]:    I0801 05:53:45.566286 281473270840000 collective_all_reduce_strategy.py:574] Check health not enabled.
[worker-3]:    INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    I0801 05:53:45.566548 281473270840000 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:35277', 'localhost:35081', 'localhost:39459', 'localhost:36055']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO
[worker-3]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-3]:    I0801 05:53:45.799512 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-1]:    I0801 05:53:45.799494 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-2]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    INFO:tensorflow:Start watcher for local signal.
[worker-2]:    I0801 05:53:45.800185 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-1]:    I0801 05:53:45.800213 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-1]:    I0801 05:53:45.800468 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-1]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    INFO:tensorflow:Start watcher for local signal.
[worker-1]:    Instructions for updating:
[worker-3]:    I0801 05:53:45.801880 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-1]:    W0801 05:53:45.800773 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-1]:    Instructions for updating:
[worker-3]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-3]:    I0801 05:53:45.802175 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-0]:    INFO:tensorflow:Start watcher for peer's signal.
[worker-1]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    I0801 05:53:45.799757 281473270840000 failure_handling.py:634] Start watcher for peer's signal.
[worker-3]:    Instructions for updating:
[worker-0]:    INFO:tensorflow:Start watcher for local signal.
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    I0801 05:53:45.800832 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-0]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-3]:    W0801 05:53:45.802483 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-0]:    I0801 05:53:45.801142 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-3]:    Instructions for updating:
[worker-0]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:Start training at 38
[worker-0]:    Instructions for updating:
[worker-3]:    I0801 05:53:45.802675 281473270840000 failure_handler_test.py:197] Start training at 38
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-3]:    INFO:tensorflow:training restarted
[worker-0]:    W0801 05:53:45.801479 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-3]:    I0801 05:53:45.808398 281473270840000 failure_handler_test.py:207] training restarted
[worker-0]:    Instructions for updating:
[worker-0]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-0]:    INFO:tensorflow:Start training at 38
[worker-0]:    I0801 05:53:45.801681 281473270840000 failure_handler_test.py:197] Start training at 38
[worker-0]:    INFO:tensorflow:training restarted
[worker-0]:    I0801 05:53:45.808025 281473270840000 failure_handler_test.py:207] training restarted
[worker-1]:    INFO:tensorflow:Start training at 38
[worker-1]:    I0801 05:53:45.800965 281473270840000 failure_handler_test.py:197] Start training at 38
[worker-1]:    INFO:tensorflow:training restarted
[worker-1]:    I0801 05:53:45.807735 281473270840000 failure_handler_test.py:207] training restarted
[worker-2]:    INFO:tensorflow:Start watcher for local signal.
[worker-2]:    I0801 05:53:45.816488 281473270840000 failure_handling.py:674] Start watcher for local signal.
[worker-2]:    INFO:tensorflow:PreemptionCheckpointHandler initialized or restored.
[worker-2]:    I0801 05:53:45.816863 281473270840000 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored.
[worker-2]:    WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    W0801 05:53:45.817214 281473270840000 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version.
[worker-2]:    Instructions for updating:
[worker-2]:    Track steps using a tf.Variable saved in checkpoint instead.
[worker-2]:    INFO:tensorflow:Start training at 38
[worker-2]:    I0801 05:53:45.817411 281473270840000 failure_handler_test.py:197] Start training at 38
[worker-2]:    INFO:tensorflow:training restarted
[worker-2]:    I0801 05:53:45.828842 281473270840000 failure_handler_test.py:207] training restarted
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:45.977209 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:45.995693 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.000599 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.011934 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.118154 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.137472 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.150907 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.147380 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.235482 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.239984 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.240093 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.251780 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.335260 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.360880 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.348944 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.383182 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.476344 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.485322 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.477378 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.491389 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e559e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:46.617811 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e559e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e61b20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e63ba0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:46.618342 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e61b20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:46.618100 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e63ba0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.627036 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    WARNING:tensorflow:5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e559e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:46.626878 281473270840000 polymorphic_function.py:156] 5 out of the last 5 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e559e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.636521 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.628565 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.652271 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e55800> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e60860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    W0801 05:53:46.736355 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e55800> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    W0801 05:53:46.736802 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e60860> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e57e20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-2]:    W0801 05:53:46.741312 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e57e20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.745474 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    WARNING:tensorflow:6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e63a60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-1]:    W0801 05:53:46.741340 281473270840000 polymorphic_function.py:156] 6 out of the last 6 calls to <function call_for_each_replica.<locals>.wrapped_fn at 0xffff95e63a60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.748784 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.750365 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.770710 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 2 finished
[worker-0]:    INFO:tensorflow:epoch 2 finished
[worker-3]:    I0801 05:53:46.825975 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-0]:    I0801 05:53:46.826305 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    INFO:tensorflow:epoch 2 finished
[worker-2]:    I0801 05:53:46.828142 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-1]:    INFO:tensorflow:epoch 2 finished
[worker-1]:    I0801 05:53:46.836570 281473270840000 failure_handler_test.py:195] epoch 2 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:46.837846 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.838266 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.845438 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:46.874679 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:46.973452 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:46.990189 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.005524 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.005589 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.115585 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.117394 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.121390 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.115278 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.213227 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.217822 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.214717 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.247003 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.427080 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.428472 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.440794 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.460493 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.679421 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.700661 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.701725 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.726795 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.811371 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.840265 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.826993 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.841009 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:47.953243 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:47.953443 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:47.954771 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:47.968571 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.031872 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.032027 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.035962 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.037020 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.113498 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.115498 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.137217 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.150578 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.235191 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.239792 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.257924 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.281294 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.385705 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.399683 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.410697 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.416328 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.553440 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.560427 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.567284 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.600613 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.695669 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.695471 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.701493 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.714450 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.798794 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.795087 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.815180 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.800091 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 3 finished
[worker-2]:    I0801 05:53:48.898397 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-3]:    INFO:tensorflow:epoch 3 finished
[worker-3]:    I0801 05:53:48.900142 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-1]:    INFO:tensorflow:epoch 3 finished
[worker-1]:    I0801 05:53:48.908846 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-0]:    INFO:tensorflow:epoch 3 finished
[worker-0]:    I0801 05:53:48.924549 281473270840000 failure_handler_test.py:195] epoch 3 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:48.930756 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:48.936417 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:48.940517 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:48.940496 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.070358 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.071232 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.075117 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.070517 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.175069 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.179414 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.212834 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.233950 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.316027 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.326246 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.340671 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.326313 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.475769 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.466082 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.481547 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.502388 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.589861 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.590606 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.620673 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.620779 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.684851 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.685591 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.700797 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.685669 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.770641 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.781116 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.785609 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.777342 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.848962 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.866164 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.867219 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.877078 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:49.955768 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:49.962056 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:49.971259 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:49.985825 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.080888 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.080247 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.101374 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.102342 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.184406 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.197426 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.200740 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.200754 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.298750 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.311809 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.311916 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.298876 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.375957 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.380667 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.383378 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.404841 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.483144 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.476001 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.503252 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.487046 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 4 finished
[worker-0]:    INFO:tensorflow:epoch 4 finished
[worker-3]:    I0801 05:53:50.573643 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-0]:    I0801 05:53:50.573970 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    INFO:tensorflow:epoch 4 finished
[worker-1]:    I0801 05:53:50.575446 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-2]:    INFO:tensorflow:epoch 4 finished
[worker-2]:    I0801 05:53:50.581057 281473270840000 failure_handler_test.py:195] epoch 4 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.584850 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.590155 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.583640 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.584248 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.649725 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.649718 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.650413 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.649550 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.827558 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.830832 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.823449 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.844302 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.911976 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:50.916648 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.920235 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.912304 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:50.980850 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:50.981199 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:50.980931 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.001349 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.085949 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.089278 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.102974 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.124850 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.219928 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.220437 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.215998 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.227957 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.300153 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.305327 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.317252 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.317590 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.404588 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.404914 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.412661 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.412694 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.475953 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.475966 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.476698 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.476264 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.540301 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.540659 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.541148 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.540423 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.605227 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.607514 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.607465 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.610740 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.680409 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.680425 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.681435 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.685836 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.761212 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.761699 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.763896 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.763978 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.837331 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.837327 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.837740 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.839337 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 5 finished
[worker-0]:    INFO:tensorflow:epoch 5 finished
[worker-0]:    I0801 05:53:51.897999 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-2]:    INFO:tensorflow:epoch 5 finished
[worker-1]:    INFO:tensorflow:epoch 5 finished
[worker-3]:    I0801 05:53:51.897663 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-1]:    I0801 05:53:51.898450 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-2]:    I0801 05:53:51.898364 281473270840000 failure_handler_test.py:195] epoch 5 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.908519 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.908581 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.908878 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.908677 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:51.984770 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:51.987057 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:51.988193 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:51.988699 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.058321 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.058850 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.059441 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.058894 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.128808 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.128691 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.132437 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.132494 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.206590 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.207423 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.222652 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.226740 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.293774 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.298075 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.295463 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.299752 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.362290 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.362301 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.366163 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.381179 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.450885 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.450680 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.450679 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.451232 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.553531 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.558054 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.560244 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.571319 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.661401 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.665201 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.665447 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.673949 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.742464 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.745257 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.745251 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.750906 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.813863 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.813726 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.814459 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.875069 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.876094 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.935060 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.935993 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.814704 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:52.993910 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.875673 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:52.994580 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:52.875621 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.936291 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 6 finished
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.043683 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-2]:    I0801 05:53:52.936357 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:52.994841 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.053519 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:epoch 6 finished
[worker-2]:    I0801 05:53:52.994787 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.044070 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 6 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.110299 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.044102 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-1]:    I0801 05:53:53.053860 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.053323 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.110585 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.110476 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.217013 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.217340 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:epoch 6 finished
[worker-0]:    I0801 05:53:53.043855 281473270840000 failure_handler_test.py:195] epoch 6 finished
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.052751 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.110577 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.216825 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.271361 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.346849 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.346817 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.349037 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.357047 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.426759 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.426771 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.433367 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.452375 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:53:54.517864 281473568242368 multi_process_runner.py:646] worker-0 exit code: 0
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:54:47.137825 281473568242368 multi_process_runner.py:646] worker-1 exit code: 0
I0801 05:54:47.141042 281473568242368 multi_process_runner.py:646] worker-2 exit code: 0
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:54:47.141297 281473568242368 multi_process_runner.py:646] worker-3 exit code: 0
[worker-0]:    I0801 05:53:53.519248 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.520103 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.520514 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.520148 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.580174 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.580776 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
I0801 05:54:47.145286 281473568242368 multi_process_runner.py:662] Joining log reading threads.
[worker-1]:    I0801 05:53:53.580837 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.580807 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.640559 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.641233 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.641751 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.640929 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.700506 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.700368 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.700203 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.700147 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.760721 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.760602 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.760589 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.760524 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.821046 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.821524 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.820852 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.881298 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.820995 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.881099 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.941803 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.881015 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.940986 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.881827 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:53.997467 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.940955 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    I0801 05:53:53.997414 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.941033 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.052867 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    I0801 05:53:53.997701 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:epoch 7 finished
[worker-3]:    I0801 05:53:54.052627 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-0]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-1]:    I0801 05:53:53.997586 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.099237 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-0]:    I0801 05:53:54.052597 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-3]:    INFO:tensorflow:epoch 7 finished
[worker-1]:    INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    INFO:tensorflow:Training finished.
[worker-0]:    INFO:tensorflow:epoch 7 finished
[worker-3]:    I0801 05:53:54.098864 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-1]:    I0801 05:53:54.052769 281473270840000 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1
[worker-2]:    I0801 05:53:54.100103 281473270840000 failure_handler_test.py:245] Training finished.
[worker-0]:    I0801 05:53:54.099038 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-3]:    INFO:tensorflow:Training finished.
[worker-1]:    INFO:tensorflow:epoch 7 finished
[worker-0]:    INFO:tensorflow:Training finished.
[worker-3]:    I0801 05:53:54.099683 281473270840000 failure_handler_test.py:245] Training finished.
[worker-1]:    I0801 05:53:54.099309 281473270840000 failure_handler_test.py:195] epoch 7 finished
[worker-0]:    I0801 05:53:54.099751 281473270840000 failure_handler_test.py:245] Training finished.
[worker-1]:    INFO:tensorflow:Training finished.
[worker-1]:    I0801 05:53:54.100209 281473270840000 failure_handler_test.py:245] Training finished.
I0801 05:54:47.163260 281473568242368 multi_process_runner.py:665] Joined log reading threads.
INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 68.05s
I0801 05:54:47.404833 281473568242368 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 68.05s
[       OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker
======================================================================
ERROR: test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest)
PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker
test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker(api_wrapping_train=False, input_arg='manager', strategy_option='MWMS_multi_worker')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test
    return test_method(self, **testcase_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated
    execute_test_method()
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator
    test_method(self, **kwargs)
  File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing
    os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM)
ProcessLookupError: [Errno 3] No such process

----------------------------------------------------------------------
Ran 4 tests in 96.577s

FAILED (errors=1)
================================================================================
//tensorflow/c:c_api_experimental_test                                   PASSED in 28.7s
//tensorflow/c:c_api_function_test                                       PASSED in 39.3s
//tensorflow/c:c_api_test_cpu                                            PASSED in 49.2s
//tensorflow/c:c_test                                                    PASSED in 43.2s
//tensorflow/c:env_test_cpu                                              PASSED in 31.3s
//tensorflow/c:kernels_test_cpu                                          PASSED in 43.5s
//tensorflow/c:ops_test                                                  PASSED in 25.5s
//tensorflow/c:tf_status_helper_test                                     PASSED in 0.1s
//tensorflow/c:while_loop_test                                           PASSED in 35.3s
//tensorflow/c/eager:c_api_cluster_test_cpu                              PASSED in 31.5s
//tensorflow/c/eager:c_api_remote_function_test_cpu                      PASSED in 32.0s
//tensorflow/c/eager:c_api_remote_test_cpu                               PASSED in 41.2s
//tensorflow/c/eager:c_api_test_cpu                                      PASSED in 37.8s
//tensorflow/c/eager:custom_device_test                                  PASSED in 42.4s
//tensorflow/c/eager/parallel_device:parallel_device_lib_test            PASSED in 51.6s
//tensorflow/c/eager/parallel_device:parallel_device_remote_test         PASSED in 35.3s
//tensorflow/c/eager/parallel_device:parallel_device_test                PASSED in 35.7s
//tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.1s
//tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 3.0s
//tensorflow/c/experimental/grappler:grappler_test                       PASSED in 35.1s
//tensorflow/c/experimental/next_pluggable_device:tensor_pjrt_buffer_util_test PASSED in 7.7s
//tensorflow/c/experimental/ops/gen/common:case_format_test              PASSED in 0.7s
//tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test               PASSED in 0.7s
//tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test          PASSED in 0.6s
//tensorflow/c/experimental/saved_model/core:constant_loading_test       PASSED in 12.6s
//tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 15.4s
//tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 12.7s
//tensorflow/c/experimental/saved_model/core:signature_flattening_test   PASSED in 21.1s
//tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 16.0s
//tensorflow/c/experimental/saved_model/core/ops:restore_ops_test        PASSED in 16.0s
//tensorflow/c/experimental/saved_model/core/ops:variable_ops_test       PASSED in 17.2s
//tensorflow/c/experimental/saved_model/internal:saved_model_api_test    PASSED in 32.7s
//tensorflow/c/experimental/stream_executor:stream_executor_test         PASSED in 0.7s
//tensorflow/c/kernels:bitcast_op_test                                   PASSED in 0.9s
//tensorflow/c/kernels:summary_op_benchmark_test                         PASSED in 0.8s
//tensorflow/c/kernels:summary_op_test                                   PASSED in 0.7s
//tensorflow/c/kernels:tensor_shape_utils_test                           PASSED in 0.2s
//tensorflow/cc:cc_op_gen_test                                           PASSED in 0.7s
//tensorflow/cc:client_client_session_test                               PASSED in 2.5s
//tensorflow/cc:coordinator_test                                         PASSED in 5.3s
//tensorflow/cc:framework_cc_ops_test                                    PASSED in 2.0s
//tensorflow/cc:framework_gradient_checker_test                          PASSED in 4.1s
//tensorflow/cc:framework_gradients_test                                 PASSED in 4.9s
//tensorflow/cc:framework_scope_test                                     PASSED in 0.5s
//tensorflow/cc:framework_while_gradients_test                           PASSED in 2.5s
//tensorflow/cc:gradients_array_grad_test                                PASSED in 5.6s
//tensorflow/cc:gradients_data_flow_grad_test                            PASSED in 2.9s
//tensorflow/cc:gradients_functional_grad_test                           PASSED in 2.6s
//tensorflow/cc:gradients_image_grad_test                                PASSED in 6.9s
//tensorflow/cc:gradients_linalg_grad_test                               PASSED in 3.2s
//tensorflow/cc:gradients_manip_grad_test                                PASSED in 2.9s
//tensorflow/cc:gradients_math_grad_test                                 PASSED in 10.3s
//tensorflow/cc:gradients_nn_grad_test                                   PASSED in 5.0s
//tensorflow/cc:gradients_resource_variable_grad_test                    PASSED in 2.6s
//tensorflow/cc:ops_const_op_test                                        PASSED in 1.3s
//tensorflow/cc:ops_while_loop_test                                      PASSED in 4.9s
//tensorflow/cc:queue_runner_test                                        PASSED in 12.6s
//tensorflow/cc/experimental/base/tests:tensor_test                      PASSED in 0.2s
//tensorflow/cc/experimental/base/tests:tensorhandle_test                PASSED in 53.4s
//tensorflow/cc/experimental/libexport:load_test                         PASSED in 0.2s
//tensorflow/cc/experimental/libexport:save_test                         PASSED in 0.1s
//tensorflow/cc/experimental/libtf:libtf_module_test                     PASSED in 36.2s
//tensorflow/cc/experimental/libtf:libtf_object_test                     PASSED in 0.2s
//tensorflow/cc/experimental/libtf:libtf_perf_test                       PASSED in 0.1s
//tensorflow/cc/experimental/libtf:libtf_runtime_test                    PASSED in 37.9s
//tensorflow/cc/experimental/libtf:libtf_transform_test                  PASSED in 38.5s
//tensorflow/cc/experimental/libtf:libtf_value_test                      PASSED in 0.1s
//tensorflow/cc/experimental/libtf:libtf_visit_test                      PASSED in 0.1s
//tensorflow/cc/experimental/libtf/impl:iostream_test                    PASSED in 0.1s
//tensorflow/cc/experimental/libtf/impl:none_test                        PASSED in 0.1s
//tensorflow/cc/experimental/libtf/impl:scalars_test                     PASSED in 0.6s
//tensorflow/cc/experimental/libtf/impl:string_test                      PASSED in 0.1s
//tensorflow/cc/experimental/libtf/impl:tensor_spec_test                 PASSED in 0.1s
//tensorflow/cc/saved_model:bundle_v2_test                               PASSED in 0.1s
//tensorflow/cc/saved_model:fingerprinting_test                          PASSED in 1.6s
//tensorflow/cc/saved_model:metrics_test                                 PASSED in 0.2s
//tensorflow/cc/saved_model:reader_test                                  PASSED in 0.1s
//tensorflow/cc/saved_model:saved_model_bundle_lite_test                 PASSED in 8.6s
//tensorflow/cc/saved_model:saved_model_bundle_test                      PASSED in 7.2s
//tensorflow/cc/saved_model:util_test                                    PASSED in 0.1s
//tensorflow/cc/saved_model/experimental/tests:saved_model_api_test      PASSED in 38.6s
//tensorflow/cc/tools:freeze_saved_model_test                            PASSED in 2.8s
//tensorflow/compiler/aot:codegen_test                                   PASSED in 32.8s
//tensorflow/compiler/jit:compilability_check_util_test                  PASSED in 22.5s
//tensorflow/compiler/jit:deadness_analysis_test                         PASSED in 10.9s
//tensorflow/compiler/jit:device_compilation_cache_test                  PASSED in 5.6s
//tensorflow/compiler/jit:device_compilation_cluster_signature_test      PASSED in 6.3s
//tensorflow/compiler/jit:device_compilation_profiler_test               PASSED in 24.9s
//tensorflow/compiler/jit:device_compiler_client_test                    PASSED in 5.4s
//tensorflow/compiler/jit:device_compiler_disable_test                   PASSED in 21.3s
//tensorflow/compiler/jit:device_executable_persistor_test               PASSED in 21.2s
//tensorflow/compiler/jit:device_util_test                               PASSED in 6.1s
//tensorflow/compiler/jit:encapsulate_util_test                          PASSED in 1.0s
//tensorflow/compiler/jit:node_matchers_test                             PASSED in 0.5s
//tensorflow/compiler/jit:resource_operation_safety_analysis_test        PASSED in 9.6s
//tensorflow/compiler/jit:shape_inference_test                           PASSED in 0.9s
//tensorflow/compiler/jit:xla_activity_listener_test                     PASSED in 21.3s
//tensorflow/compiler/jit:xla_cluster_util_test                          PASSED in 14.0s
//tensorflow/compiler/jit:xla_compile_util_test                          PASSED in 5.7s
//tensorflow/compiler/jit:xla_kernel_creator_test                        PASSED in 13.2s
//tensorflow/compiler/jit:xla_launch_util_test                           PASSED in 25.8s
//tensorflow/compiler/jit/tests:auto_clustering_test                     PASSED in 31.1s
//tensorflow/compiler/mlir:mlir_graph_optimization_pass_test             PASSED in 13.9s
//tensorflow/compiler/mlir:register_common_dialects_test                 PASSED in 16.2s
//tensorflow/compiler/mlir/lite:lstm_utils_test                          PASSED in 1.4s
//tensorflow/compiler/mlir/lite:perception_ops_utils_test                PASSED in 0.7s
//tensorflow/compiler/mlir/lite:size_utils_test                          PASSED in 0.1s
//tensorflow/compiler/mlir/lite:tftext_utils_test                        PASSED in 0.5s
//tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test   PASSED in 1.1s
//tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 8.0s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 2.6s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test        PASSED in 0.3s
//tensorflow/compiler/mlir/lite/quantization:numerical_utils_test        PASSED in 0.1s
//tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test    PASSED in 13.6s
//tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test  PASSED in 10.2s
//tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test             PASSED in 2.4s
//tensorflow/compiler/mlir/lite/stablehlo/tests:compose-uniform-quantized-type.mlir.test PASSED in 17.4s
//tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 2.4s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 2.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 1.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test       PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-reduce-window.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 2.5s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-subtract.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/stablehlo/tests:uniform-quantized-stablehlo-to-tfl.mlir.test PASSED in 2.5s
//tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test        PASSED in 1.0s
//tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test             PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test               PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test     PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test             PASSED in 1.8s
//tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test              PASSED in 1.0s
//tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test     PASSED in 3.9s
//tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:inlining.mlir.test                 PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test      PASSED in 2.4s
//tensorflow/compiler/mlir/lite/tests:legalize-tensorlist.mlir.test      PASSED in 1.9s
//tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test       PASSED in 3.4s
//tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test   PASSED in 2.6s
//tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test    PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test        PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test              PASSED in 1.4s
//tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test      PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test     PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test          PASSED in 1.0s
//tensorflow/compiler/mlir/lite/tests:ops.mlir.test                      PASSED in 1.3s
//tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests:optimize.mlir.test                 PASSED in 2.6s
//tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test  PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test       PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test        PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test            PASSED in 1.6s
//tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.8s
//tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 2.9s
//tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test  PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test         PASSED in 3.2s
//tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test    PASSED in 1.6s
//tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test               PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test   PASSED in 3.4s
//tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test  PASSED in 2.0s
//tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test       PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests:quantize.mlir.test                 PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test         PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test    PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test          PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test    PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test        PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test        PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test        PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 1.5s
//tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test             PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test         PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test    PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test    PASSED in 1.2s
//tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 21.9s
//tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.2s
//tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 2.0s
//tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 1.2s
//tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 1.9s
//tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test           PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test     PASSED in 2.6s
//tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 2.1s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants_offset.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op_offset.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test    PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 1.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test     PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test     PASSED in 55.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test     PASSED in 2.3s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test   PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 1.2s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test  PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 2.0s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test  PASSED in 0.5s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test   PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 3.9s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test   PASSED in 1.0s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_tensorlist_reserve.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 2.0s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 1.8s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test    PASSED in 1.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test  PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test     PASSED in 25.5s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test     PASSED in 0.9s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test   PASSED in 1.2s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test   PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test       PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test  PASSED in 1.5s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test   PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test     PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test  PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 2.0s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test PASSED in 9.3s
//tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_types_test PASSED in 21.0s
//tensorflow/compiler/mlir/quantization/stablehlo:math_utils_test        PASSED in 0.1s
//tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 4.5s
//tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.6s
//tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 45.8s
//tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.3s
//tensorflow/compiler/mlir/quantization/tensorflow/cc:constant_fold_test PASSED in 41.7s
//tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 5.4s
//tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.4s
//tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test  PASSED in 0.2s
//tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.1s
//tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 76.1s
//tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 52.1s
//tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 10.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 2.0s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 2.6s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_hashtable_ops_as_args.mlir.test PASSED in 20.2s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 1.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 2.2s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_duplicate_resource_ops.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 54.4s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 27.3s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 3.2s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 14.9s
//tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 37.7s
//tensorflow/compiler/mlir/stablehlo:stablehlo_test                      PASSED in 0.1s
//tensorflow/compiler/mlir/tensorflow:bridge_logger_test                 PASSED in 7.9s
//tensorflow/compiler/mlir/tensorflow:call_graph_util_test               PASSED in 0.5s
//tensorflow/compiler/mlir/tensorflow:cluster_util_test                  PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow:convert_tensor_test                PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow:convert_type_test                  PASSED in 0.2s
//tensorflow/compiler/mlir/tensorflow:device_util_test                   PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow:dump_graph_test                    PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test                PASSED in 15.2s
//tensorflow/compiler/mlir/tensorflow:error_util_test                    PASSED in 0.1s
//tensorflow/compiler/mlir/tensorflow:tf_mlir_translate_registration_test PASSED in 14.6s
//tensorflow/compiler/mlir/tensorflow:tf_saved_model_test                PASSED in 0.3s
//tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test       PASSED in 0.5s
//tensorflow/compiler/mlir/tensorflow:xla_rewrite_util_test              PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 1.8s
//tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test    PASSED in 2.1s
//tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test       PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test  PASSED in 1.8s
//tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test  PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test      PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 11.3s
//tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test  PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test        PASSED in 1.5s
//tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test             PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:embedding_program_key.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:embedding_sequencing.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test         PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 2.3s
//tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test     PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test   PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test  PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test          PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test   PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test         PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test      PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test   PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test           PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test     PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test   PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:legalize_hlo.mlir.test       PASSED in 1.8s
//tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test       PASSED in 2.7s
//tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test    PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test           PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test          PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test           PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test   PASSED in 2.1s
//tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 3.3s
//tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 2.2s
//tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test  PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test  PASSED in 1.5s
//tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 3.4s
//tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test    PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test      PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 2.5s
//tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test     PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 2.1s
//tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 3.3s
//tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test             PASSED in 2.6s
//tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 2.0s
//tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test      PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test    PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test        PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_asset_sinking.mlir.test PASSED in 2.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 2.0s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 2.0s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test     PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test     PASSED in 2.0s
//tensorflow/compiler/mlir/tensorflow/tests:tfrt_ops.mlir.test           PASSED in 3.9s
//tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 2.3s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test        PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test       PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 2.4s
//tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test         PASSED in 1.8s
//tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test        PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite_v2.mlir.test     PASSED in 2.2s
//tensorflow/compiler/mlir/tensorflow/tests:xla_sharding_util_test       PASSED in 0.4s
//tensorflow/compiler/mlir/tensorflow/tests:xla_validate_iputs.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:convert_mhlo_quant_to_int.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 2.8s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 21.7s
//tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 2.2s
//tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.3s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 1.5s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 1.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 6.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 2.2s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 2.4s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 1.5s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 2.3s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 2.6s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 3.0s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graph:convert_tensor.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 1.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 1.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tf2xla/api/v0:compile_mlir_util_test          PASSED in 4.6s
//tensorflow/compiler/mlir/tf2xla/api/v0:compile_tf_graph_test           PASSED in 0.3s
//tensorflow/compiler/mlir/tf2xla/api/v1:legalize_tf_test                PASSED in 26.4s
//tensorflow/compiler/mlir/tf2xla/internal:mlir_pass_instrumentation_test PASSED in 6.7s
//tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test          PASSED in 0.8s
//tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.8s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 2.4s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-no-tf2xla-fallback.mlir.test PASSED in 6.1s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 2.4s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test            PASSED in 8.7s
//tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tf2xla/transforms:legalization_op_config_test PASSED in 34.6s
//tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test        PASSED in 20.2s
//tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 18.8s
//tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test   PASSED in 0.5s
//tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test        PASSED in 1.9s
//tensorflow/compiler/mlir/tfr:graph_decompose_test                      PASSED in 13.2s
//tensorflow/compiler/mlir/tfr:node_expansion_test                       PASSED in 12.2s
//tensorflow/compiler/mlir/tfr:op_reg_gen_test                           PASSED in 27.1s
//tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test                    PASSED in 5.3s
//tensorflow/compiler/mlir/tfr:tfr_gen_test                              PASSED in 27.1s
//tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test      PASSED in 31.5s
//tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test                 PASSED in 30.0s
//tensorflow/compiler/mlir/tfrt/tests:batch_function_fallback_resource_variable_as_captured_tensor.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests:batch_function_lowering.mlir.test  PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests:convert_ref_variables.mlir.test    PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests:cross_device_transfer.mlir.test    PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests:deduplicate_if_results.mlir.test   PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests:fuse_tpu_compile_and_execute_ops.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops.mlir.test      PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops_mlrt.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tfrt/tests:optimize.mlir.test                 PASSED in 5.8s
//tensorflow/compiler/mlir/tfrt/tests:remove_device_attribute.mlir.test  PASSED in 1.0s
//tensorflow/compiler/mlir/tfrt/tests:sink_in_invariant_ops.mlir.test    PASSED in 1.2s
//tensorflow/compiler/mlir/tfrt/tests:xla_launch_fallback.mlir.test      PASSED in 1.4s
//tensorflow/compiler/mlir/tfrt/tests:xla_rewrite.mlir.test              PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/analysis:cost_analysis.mlir.test   PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests/analysis:tensor_array_side_effect_analysis.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests/analysis:update_op_cost_in_tfrt_mlir_test PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests/ir:fallback_opt.mlir.test          PASSED in 1.4s
//tensorflow/compiler/mlir/tfrt/tests/ir:tfrt_fallback_util_test         PASSED in 0.6s
//tensorflow/compiler/mlir/tfrt/tests/mlrt:assign_op_key.mlir.test       PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests/mlrt:fuse_mlrt_ops.mlir.test       PASSED in 2.2s
//tensorflow/compiler/mlir/tfrt/tests/mlrt:inline.mlir.test              PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/mlrt:parallelization.mlir.test     PASSED in 1.2s
//tensorflow/compiler/mlir/tfrt/tests/mlrt:tf_to_mlrt.mlir.test          PASSED in 1.0s
//tensorflow/compiler/mlir/tfrt/tests/mlrt:tpu_conversions.mlir.test     PASSED in 1.0s
//tensorflow/compiler/mlir/tfrt/tests/mlrt:while_to_map_fn.mlir.test     PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:attributes.mlir.test  PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:basic.mlir.test       PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate_failed.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:const_tensor.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:control_flow.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:decompose_resource_op.mlir.test PASSED in 2.1s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:derived_attrs.mlir.test PASSED in 2.6s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:device_conversion.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:errors.mlir.test      PASSED in 1.5s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback.mlir.test    PASSED in 1.1s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_canonicalization.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_inline.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes.mlir.test PASSED in 1.0s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes_multiple_callers.mlir.test PASSED in 2.1s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_use_fallback_tensor.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:insert_fallback_tensor_copy.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:merge_tf_if_ops.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:optimize_tf_control_flow_side_effect.mlir.test PASSED in 1.4s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:remove_tf_if_const_args.mlir.test PASSED in 1.3s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:reorder_assert.mlir.test PASSED in 5.5s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:side_effects.mlir.test PASSED in 1.2s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline.mlir.test PASSED in 2.3s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline_refvar.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:whileop.mlir.test     PASSED in 0.8s
//tensorflow/compiler/mlir/tfrt/translate/mlrt:mlir_to_bytecode_test     PASSED in 0.2s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 1.6s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test    PASSED in 0.9s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test      PASSED in 1.9s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test        PASSED in 1.5s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test          PASSED in 0.7s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.6s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test         PASSED in 0.6s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 1.1s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 2.1s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 0.8s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 0.9s
//tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_to_jit_invocations.mlir.test PASSED in 0.7s
//tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test        PASSED in 0.6s
//tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test         PASSED in 0.6s
//tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test             PASSED in 0.7s
//tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test      PASSED in 5.5s
//tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test     PASSED in 2.5s
//tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test                PASSED in 0.7s
//tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test   PASSED in 0.8s
//tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test        PASSED in 1.2s
//tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test           PASSED in 0.8s
//tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test  PASSED in 0.8s
//tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test      PASSED in 2.0s
//tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 2.6s
//tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 1.5s
//tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test     PASSED in 7.3s
//tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test   PASSED in 0.6s
//tensorflow/compiler/tests:adadelta_test_cpu                            PASSED in 18.0s
//tensorflow/compiler/tests:adagrad_da_test_cpu                          PASSED in 14.0s
//tensorflow/compiler/tests:adagrad_test_cpu                             PASSED in 19.3s
//tensorflow/compiler/tests:adam_test_cpu                                PASSED in 18.2s
//tensorflow/compiler/tests:add_n_test_cpu                               PASSED in 29.3s
//tensorflow/compiler/tests:argminmax_test_cpu                           PASSED in 16.7s
//tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test          PASSED in 27.8s
//tensorflow/compiler/tests:bucketize_op_test_cpu                        PASSED in 12.1s
//tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test       PASSED in 11.3s
//tensorflow/compiler/tests:case_test_cpu                                PASSED in 19.7s
//tensorflow/compiler/tests:cast_ops_test_cpu                            PASSED in 18.4s
//tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test           PASSED in 12.6s
//tensorflow/compiler/tests:categorical_op_test_cpu                      PASSED in 13.9s
//tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test     PASSED in 51.5s
//tensorflow/compiler/tests:cholesky_op_test_cpu                         PASSED in 33.4s
//tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test        PASSED in 17.7s
//tensorflow/compiler/tests:clustering_test_cpu                          PASSED in 26.0s
//tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test         PASSED in 23.8s
//tensorflow/compiler/tests:concat_ops_test_cpu                          PASSED in 12.0s
//tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test         PASSED in 12.6s
//tensorflow/compiler/tests:cond_test_cpu                                PASSED in 11.2s
//tensorflow/compiler/tests:const_arg_test_cpu                           PASSED in 10.6s
//tensorflow/compiler/tests:const_test_cpu                               PASSED in 10.3s
//tensorflow/compiler/tests:data_format_ops_test_cpu                     PASSED in 15.1s
//tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test    PASSED in 19.2s
//tensorflow/compiler/tests:dense_layer_test_cpu                         PASSED in 17.1s
//tensorflow/compiler/tests:dynamic_slice_ops_test_cpu                   PASSED in 11.9s
//tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test  PASSED in 15.1s
//tensorflow/compiler/tests:dynamic_stitch_test_cpu                      PASSED in 12.9s
//tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test     PASSED in 10.5s
//tensorflow/compiler/tests:eager_test_cpu                               PASSED in 21.8s
//tensorflow/compiler/tests:einsum_op_test_cpu                           PASSED in 10.1s
//tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test          PASSED in 11.9s
//tensorflow/compiler/tests:ensure_shape_op_test_cpu                     PASSED in 33.1s
//tensorflow/compiler/tests:extract_image_patches_op_test_cpu            PASSED in 10.3s
//tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 10.7s
//tensorflow/compiler/tests:fake_quant_ops_test_cpu                      PASSED in 22.4s
//tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test     PASSED in 18.0s
//tensorflow/compiler/tests:fifo_queue_test_cpu                          PASSED in 11.8s
//tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test         PASSED in 12.2s
//tensorflow/compiler/tests:ftrl_ops_test_cpu                            PASSED in 12.1s
//tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test           PASSED in 12.1s
//tensorflow/compiler/tests:function_test_cpu                            PASSED in 10.6s
//tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test           PASSED in 10.5s
//tensorflow/compiler/tests:gather_nd_op_test_cpu                        PASSED in 12.6s
//tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test       PASSED in 25.9s
//tensorflow/compiler/tests:gather_test_cpu                              PASSED in 38.2s
//tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test             PASSED in 65.5s
//tensorflow/compiler/tests:jit_test_cpu                                 PASSED in 51.6s
//tensorflow/compiler/tests:listdiff_op_test_cpu                         PASSED in 24.4s
//tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test        PASSED in 16.9s
//tensorflow/compiler/tests:lrn_ops_test_cpu                             PASSED in 41.5s
//tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test            PASSED in 10.1s
//tensorflow/compiler/tests:lstm_test_cpu                                PASSED in 25.4s
//tensorflow/compiler/tests:manip_ops_test_cpu                           PASSED in 12.8s
//tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test          PASSED in 15.8s
//tensorflow/compiler/tests:matrix_band_part_test_cpu                    PASSED in 61.6s
//tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test   PASSED in 72.4s
//tensorflow/compiler/tests:matrix_inverse_op_test_cpu                   PASSED in 20.5s
//tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test  PASSED in 21.1s
//tensorflow/compiler/tests:matrix_solve_op_test_cpu                     PASSED in 40.7s
//tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test    PASSED in 11.9s
//tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu          PASSED in 30.0s
//tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu_mlir_bridge_test PASSED in 32.5s
//tensorflow/compiler/tests:momentum_test_cpu                            PASSED in 12.9s
//tensorflow/compiler/tests:nary_ops_test_cpu                            PASSED in 24.4s
//tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test           PASSED in 12.4s
//tensorflow/compiler/tests:nullary_ops_test_cpu                         PASSED in 12.9s
//tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test        PASSED in 10.7s
//tensorflow/compiler/tests:placeholder_test_cpu                         PASSED in 10.0s
//tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test        PASSED in 13.1s
//tensorflow/compiler/tests:proximal_adagrad_test_cpu                    PASSED in 25.9s
//tensorflow/compiler/tests:proximal_gradient_descent_test_cpu           PASSED in 31.6s
//tensorflow/compiler/tests:quantized_ops_test_cpu                       PASSED in 12.3s
//tensorflow/compiler/tests:reduce_window_test_cpu                       PASSED in 13.9s
//tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test      PASSED in 12.1s
//tensorflow/compiler/tests:reshape_op_test_cpu                          PASSED in 14.4s
//tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test         PASSED in 18.8s
//tensorflow/compiler/tests:reverse_ops_test_cpu                         PASSED in 14.5s
//tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test        PASSED in 27.8s
//tensorflow/compiler/tests:reverse_sequence_op_test_cpu                 PASSED in 10.7s
//tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 11.1s
//tensorflow/compiler/tests:rmsprop_test_cpu                             PASSED in 13.6s
//tensorflow/compiler/tests:scatter_nd_op_test_cpu                       PASSED in 24.2s
//tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test      PASSED in 37.2s
//tensorflow/compiler/tests:searchsorted_op_test_cpu                     PASSED in 18.6s
//tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test    PASSED in 12.3s
//tensorflow/compiler/tests:segment_reduction_ops_test_cpu               PASSED in 31.8s
//tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 31.2s
//tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu                 PASSED in 18.3s
//tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 29.2s
//tensorflow/compiler/tests:slice_ops_test_cpu                           PASSED in 19.5s
//tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test          PASSED in 30.0s
//tensorflow/compiler/tests:sparse_to_dense_op_test_cpu                  PASSED in 10.4s
//tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 10.3s
//tensorflow/compiler/tests:stack_ops_test_cpu                           PASSED in 11.0s
//tensorflow/compiler/tests:tensor_float_32_test_cpu                     PASSED in 14.4s
//tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test    PASSED in 18.0s
//tensorflow/compiler/tests:tensor_list_ops_test_cpu                     PASSED in 15.2s
//tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu              PASSED in 25.5s
//tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 42.9s
//tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu               PASSED in 15.3s
//tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 18.3s
//tensorflow/compiler/tests:unique_ops_test_cpu                          PASSED in 10.2s
//tensorflow/compiler/tests:variable_ops_test_cpu                        PASSED in 29.8s
//tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test       PASSED in 18.5s
//tensorflow/compiler/tests:where_op_test_cpu                            PASSED in 12.0s
//tensorflow/compiler/tests:while_test_cpu                               PASSED in 14.0s
//tensorflow/compiler/tests:xla_call_module_no_platform_check_test_cpu   PASSED in 14.2s
//tensorflow/compiler/tests:xla_call_module_no_shape_assertions_check_test_cpu PASSED in 11.4s
//tensorflow/compiler/tests:xla_call_module_test_cpu                     PASSED in 35.9s
//tensorflow/compiler/tests:xla_custom_call_ops_test_cpu                 PASSED in 9.4s
//tensorflow/compiler/tests:xla_device_gpu_test_cpu                      PASSED in 11.1s
//tensorflow/compiler/tests:xla_device_test_cpu                          PASSED in 16.8s
//tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test         PASSED in 18.1s
//tensorflow/compiler/tests:xla_ops_test_cpu                             PASSED in 36.1s
//tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test            PASSED in 44.8s
//tensorflow/compiler/tests:xla_test_test                                PASSED in 13.2s
//tensorflow/compiler/tf2xla:const_analysis_test                         PASSED in 5.6s
//tensorflow/compiler/tf2xla:cpu_function_runtime_test                   PASSED in 0.3s
//tensorflow/compiler/tf2xla:functionalize_cond_test                     PASSED in 0.6s
//tensorflow/compiler/tf2xla:functionalize_control_flow_test             PASSED in 0.8s
//tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu      PASSED in 27.8s
//tensorflow/compiler/tf2xla:graph_compiler_test                         PASSED in 5.8s
//tensorflow/compiler/tf2xla:literal_util_test                           PASSED in 0.5s
//tensorflow/compiler/tf2xla:resource_operation_table_test               PASSED in 5.9s
//tensorflow/compiler/tf2xla:resource_util_test_cpu                      PASSED in 1.9s
//tensorflow/compiler/tf2xla:sharding_util_test                          PASSED in 0.6s
//tensorflow/compiler/tf2xla:tf2xla_opset_test                           PASSED in 8.6s
//tensorflow/compiler/tf2xla:tf2xla_test                                 PASSED in 18.1s
//tensorflow/compiler/tf2xla:tf2xla_util_test                            PASSED in 1.2s
//tensorflow/compiler/tf2xla:xla_compiler_test                           PASSED in 16.9s
//tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test          PASSED in 20.2s
//tensorflow/compiler/tf2xla:xla_op_registry_test                        PASSED in 5.1s
//tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test            PASSED in 1.4s
//tensorflow/compiler/xla:array2d_test                                   PASSED in 0.2s
//tensorflow/compiler/xla:array3d_test                                   PASSED in 0.2s
//tensorflow/compiler/xla:array4d_test                                   PASSED in 0.4s
//tensorflow/compiler/xla:array_test                                     PASSED in 0.1s
//tensorflow/compiler/xla:bit_cast_test                                  PASSED in 0.1s
//tensorflow/compiler/xla:comparison_util_test                           PASSED in 0.2s
//tensorflow/compiler/xla:debug_options_parsers_test                     PASSED in 0.1s
//tensorflow/compiler/xla:index_util_test                                PASSED in 0.3s
//tensorflow/compiler/xla:iterator_util_test                             PASSED in 0.1s
//tensorflow/compiler/xla:layout_test                                    PASSED in 0.2s
//tensorflow/compiler/xla:layout_util_test                               PASSED in 0.1s
//tensorflow/compiler/xla:literal_test                                   PASSED in 0.5s
//tensorflow/compiler/xla:parse_flags_from_env_test                      PASSED in 0.5s
//tensorflow/compiler/xla:permutation_util_test                          PASSED in 0.1s
//tensorflow/compiler/xla:primitive_util_test                            PASSED in 0.6s
//tensorflow/compiler/xla:refcounting_hash_map_test                      PASSED in 0.1s
//tensorflow/compiler/xla:reference_util_test                            PASSED in 0.2s
//tensorflow/compiler/xla:shape_test                                     PASSED in 0.1s
//tensorflow/compiler/xla:shape_tree_test                                PASSED in 0.1s
//tensorflow/compiler/xla:shape_util_test                                PASSED in 2.4s
//tensorflow/compiler/xla:status_macros_test                             PASSED in 0.2s
//tensorflow/compiler/xla:text_literal_reader_test                       PASSED in 1.0s
//tensorflow/compiler/xla:text_literal_writer_test                       PASSED in 0.1s
//tensorflow/compiler/xla:types_test                                     PASSED in 0.1s
//tensorflow/compiler/xla:util_test                                      PASSED in 0.1s
//tensorflow/compiler/xla:window_util_test                               PASSED in 0.1s
//tensorflow/compiler/xla/client:padding_test                            PASSED in 0.6s
//tensorflow/compiler/xla/client:xla_builder_test                        PASSED in 0.3s
//tensorflow/compiler/xla/client/lib:arithmetic_test_cpu                 PASSED in 8.4s
//tensorflow/compiler/xla/client/lib:comparators_test_cpu                PASSED in 8.8s
//tensorflow/compiler/xla/client/lib:constants_test_cpu                  PASSED in 27.7s
//tensorflow/compiler/xla/client/lib:logdet_test_cpu                     PASSED in 9.4s
//tensorflow/compiler/xla/client/lib:math_test_cpu                       PASSED in 13.0s
//tensorflow/compiler/xla/client/lib:matrix_test_cpu                     PASSED in 11.4s
//tensorflow/compiler/xla/client/lib:pooling_test_cpu                    PASSED in 7.6s
//tensorflow/compiler/xla/client/lib:qr_test_cpu                         PASSED in 12.3s
//tensorflow/compiler/xla/client/lib:slicing_test_cpu                    PASSED in 9.6s
//tensorflow/compiler/xla/client/lib:sorting_test_cpu                    PASSED in 7.5s
//tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test           PASSED in 8.7s
//tensorflow/compiler/xla/experiments/sm_bandwidth_benchmark:sm_bw_test  PASSED in 0.2s
//tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test               PASSED in 40.9s
//tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_solver_test PASSED in 0.9s
//tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_test PASSED in 3.7s
//tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test      PASSED in 0.8s
//tensorflow/compiler/xla/hlo/utils:hlo_live_range_test                  PASSED in 1.3s
//tensorflow/compiler/xla/hlo/utils:hlo_matchers_test                    PASSED in 1.4s
//tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test               PASSED in 0.2s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test PASSED in 1.8s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test PASSED in 1.1s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test PASSED in 54.2s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test PASSED in 7.7s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test PASSED in 1.3s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:stream_assignment.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test PASSED in 1.9s
//tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test   PASSED in 0.7s
//tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test            PASSED in 0.7s
//tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test     PASSED in 0.7s
//tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test        PASSED in 1.0s
//tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.2s
//tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test    PASSED in 0.1s
//tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test PASSED in 1.3s
//tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test PASSED in 2.4s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test PASSED in 1.8s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test PASSED in 2.4s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test  PASSED in 0.6s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test  PASSED in 1.9s
//tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test PASSED in 3.1s
//tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.4s
//tensorflow/compiler/xla/mlir/utils:error_util_test                     PASSED in 0.2s
//tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test         PASSED in 1.2s
//tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test           PASSED in 0.5s
//tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test               PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/bufferization/hlo_one_shot_bufferize.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 2.0s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/analysis.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 1.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate_invalid.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_simplification.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 1.1s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collect_stats.mlir.test PASSED in 14.2s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/batch_matmul.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 2.3s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 2.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reshape_map.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 1.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 3.3s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d_map.mlir.test PASSED in 1.4s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 1.4s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 1.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test    PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/optimize_linalg_ops.mlir.test PASSED in 1.1s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/rewrite_forall_to_for.mlir.test PASSED in 1.4s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 1.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test      PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test    PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 2.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 2.3s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 1.2s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 1.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 2.0s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 1.1s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-rng-to-linalg.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 4.3s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 1.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-torch-index-select-to-gather.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 2.2s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test  PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 1.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 1.2s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 1.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test      PASSED in 1.3s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 2.1s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 2.1s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 1.3s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 1.5s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test  PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test      PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test   PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test          PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test        PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test             PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test    PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test   PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test      PASSED in 1.3s
//tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 1.0s
//tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test   PASSED in 0.9s
//tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test         PASSED in 1.3s
//tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test  PASSED in 1.2s
//tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test         PASSED in 0.6s
//tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test            PASSED in 0.7s
//tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test           PASSED in 0.8s
//tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test          PASSED in 0.5s
//tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.1s
//tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.1s
//tensorflow/compiler/xla/pjrt:host_callback_test                        PASSED in 0.2s
//tensorflow/compiler/xla/pjrt:lru_cache_test                            PASSED in 0.8s
//tensorflow/compiler/xla/pjrt:pjrt_api_test                             PASSED in 0.7s
//tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu                      PASSED in 9.2s
//tensorflow/compiler/xla/pjrt:pjrt_compiler_test                        PASSED in 0.2s
//tensorflow/compiler/xla/pjrt:pjrt_executable_test                      PASSED in 0.2s
//tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test          PASSED in 12.5s
//tensorflow/compiler/xla/pjrt:semaphore_test                            PASSED in 0.1s
//tensorflow/compiler/xla/pjrt:tf_pjrt_client_test                       PASSED in 11.4s
//tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test                 PASSED in 15.6s
//tensorflow/compiler/xla/pjrt:tracked_device_buffer_test                PASSED in 7.9s
//tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test       PASSED in 0.2s
//tensorflow/compiler/xla/pjrt:transpose_test                            PASSED in 53.3s
//tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test                     PASSED in 15.5s
//tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test                 PASSED in 1.3s
//tensorflow/compiler/xla/pjrt/distributed:client_server_test            PASSED in 43.9s
//tensorflow/compiler/xla/pjrt/distributed:topology_util_test            PASSED in 0.1s
//tensorflow/compiler/xla/python:outfeed_receiver_test_cpu               PASSED in 8.0s
//tensorflow/compiler/xla/python:xplane_to_profile_instructions_test     PASSED in 0.9s
//tensorflow/compiler/xla/python/ifrt:array_test                         PASSED in 0.2s
//tensorflow/compiler/xla/python/ifrt:array_test_no_impl                 PASSED in 0.5s
//tensorflow/compiler/xla/python/ifrt:client_test_no_impl                PASSED in 0.4s
//tensorflow/compiler/xla/python/ifrt:future_test                        PASSED in 0.3s
//tensorflow/compiler/xla/python/ifrt:index_domain_test                  PASSED in 0.4s
//tensorflow/compiler/xla/python/ifrt:index_test                         PASSED in 0.2s
//tensorflow/compiler/xla/python/ifrt:memory_test                        PASSED in 0.5s
//tensorflow/compiler/xla/python/ifrt:serdes_test                        PASSED in 0.4s
//tensorflow/compiler/xla/python/ifrt:shape_test                         PASSED in 0.2s
//tensorflow/compiler/xla/python/ifrt:sharding_serdes_test               PASSED in 0.4s
//tensorflow/compiler/xla/python/ifrt:sharding_test                      PASSED in 1.0s
//tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl                 PASSED in 0.2s
//tensorflow/compiler/xla/python/ifrt/ir/tests:executable_test_no_impl   PASSED in 2.9s
//tensorflow/compiler/xla/python/ifrt/ir/tests:ifrt_duplicated_callee_elimination.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_interface_verification.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test    PASSED in 1.0s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test PASSED in 1.9s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test    PASSED in 0.7s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test     PASSED in 0.6s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test PASSED in 1.1s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test  PASSED in 0.8s
//tensorflow/compiler/xla/python/ifrt/support:sharding_param_to_op_sharding_test PASSED in 0.7s
//tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 19.8s
//tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 10.2s
//tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 5.7s
//tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 8.4s
//tensorflow/compiler/xla/python/pjrt_ifrt:xla_executable_test_no_impl   PASSED in 1.3s
//tensorflow/compiler/xla/python/pjrt_ifrt:xla_program_serdes_test       PASSED in 1.5s
//tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_serdes_test      PASSED in 0.3s
//tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_test             PASSED in 9.7s
//tensorflow/compiler/xla/python_api:xla_literal_test                    PASSED in 1.7s
//tensorflow/compiler/xla/python_api:xla_shape_test                      PASSED in 2.5s
//tensorflow/compiler/xla/rpc:grpc_client_test                           PASSED in 3.5s
//tensorflow/compiler/xla/runtime:arguments_test                         PASSED in 0.2s
//tensorflow/compiler/xla/runtime:async_runtime_test                     PASSED in 0.1s
//tensorflow/compiler/xla/runtime:custom_call_test                       PASSED in 2.3s
//tensorflow/compiler/xla/runtime:diagnostics_test                       PASSED in 0.2s
//tensorflow/compiler/xla/runtime:executable_test                        PASSED in 3.1s
//tensorflow/compiler/xla/runtime:ffi_test                               PASSED in 1.5s
//tensorflow/compiler/xla/runtime:map_by_type_test                       PASSED in 0.1s
//tensorflow/compiler/xla/runtime:module_test                            PASSED in 0.1s
//tensorflow/compiler/xla/runtime:results_test                           PASSED in 0.2s
//tensorflow/compiler/xla/runtime:state_test                             PASSED in 0.1s
//tensorflow/compiler/xla/runtime:symbolic_shape_test                    PASSED in 0.2s
//tensorflow/compiler/xla/runtime:type_id_test                           PASSED in 0.1s
//tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 15.5s
//tensorflow/compiler/xla/service:algebraic_simplifier_test              PASSED in 2.8s
//tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test      PASSED in 0.7s
//tensorflow/compiler/xla/service:all_gather_combiner_test               PASSED in 0.8s
//tensorflow/compiler/xla/service:all_gather_decomposer_test             PASSED in 0.8s
//tensorflow/compiler/xla/service:all_reduce_combiner_test               PASSED in 1.0s
//tensorflow/compiler/xla/service:all_reduce_contiguous_test             PASSED in 0.8s
//tensorflow/compiler/xla/service:all_reduce_folder_test                 PASSED in 0.9s
//tensorflow/compiler/xla/service:all_reduce_promotion_test              PASSED in 0.6s
//tensorflow/compiler/xla/service:all_reduce_reassociate_test            PASSED in 0.8s
//tensorflow/compiler/xla/service:all_reduce_simplifier_test             PASSED in 0.8s
//tensorflow/compiler/xla/service:ar_crs_combiner_test                   PASSED in 1.6s
//tensorflow/compiler/xla/service:async_collective_creator_test          PASSED in 1.2s
//tensorflow/compiler/xla/service:async_op_canonicalizer_test            PASSED in 0.6s
//tensorflow/compiler/xla/service:batch_dot_simplification_test          PASSED in 1.3s
//tensorflow/compiler/xla/service:batchnorm_expander_test_cpu            PASSED in 7.5s
//tensorflow/compiler/xla/service:bfloat16_conversion_folding_test       PASSED in 0.7s
//tensorflow/compiler/xla/service:bfloat16_propagation_test              PASSED in 1.3s
//tensorflow/compiler/xla/service:bitcast_dtypes_expander_test           PASSED in 0.9s
//tensorflow/compiler/xla/service:broadcast_canonicalizer_test           PASSED in 0.9s
//tensorflow/compiler/xla/service:buffer_assignment_test                 PASSED in 7.0s
//tensorflow/compiler/xla/service:call_graph_test                        PASSED in 0.9s
//tensorflow/compiler/xla/service:call_inliner_test                      PASSED in 0.7s
//tensorflow/compiler/xla/service:change_op_data_type_test               PASSED in 0.8s
//tensorflow/compiler/xla/service:collective_ops_utils_test              PASSED in 0.2s
//tensorflow/compiler/xla/service:collective_permute_decomposer_test     PASSED in 0.8s
//tensorflow/compiler/xla/service:collective_pipeliner_test              PASSED in 1.1s
//tensorflow/compiler/xla/service:collective_transformation_reorderer_test PASSED in 0.9s
//tensorflow/compiler/xla/service:collectives_schedule_linearizer_test   PASSED in 0.9s
//tensorflow/compiler/xla/service:compilation_environments_test          PASSED in 0.9s
//tensorflow/compiler/xla/service:conditional_canonicalizer_test         PASSED in 0.7s
//tensorflow/compiler/xla/service:conditional_code_motion_test           PASSED in 0.9s
//tensorflow/compiler/xla/service:conditional_simplifier_test            PASSED in 0.8s
//tensorflow/compiler/xla/service:conditional_to_select_test             PASSED in 0.9s
//tensorflow/compiler/xla/service:constant_value_test                    PASSED in 0.2s
//tensorflow/compiler/xla/service:convert_async_collectives_to_sync_test PASSED in 2.2s
//tensorflow/compiler/xla/service:convert_mover_test                     PASSED in 0.9s
//tensorflow/compiler/xla/service:convert_operand_folding_test           PASSED in 0.7s
//tensorflow/compiler/xla/service:convolution_4d_expander_test           PASSED in 0.7s
//tensorflow/compiler/xla/service:convolution_group_converter_test       PASSED in 1.3s
//tensorflow/compiler/xla/service:convolution_pred_expander_test         PASSED in 1.7s
//tensorflow/compiler/xla/service:copy_insertion_test                    PASSED in 2.1s
//tensorflow/compiler/xla/service:custom_call_status_test                PASSED in 0.1s
//tensorflow/compiler/xla/service:defuser_test                           PASSED in 0.8s
//tensorflow/compiler/xla/service:despecializer_test                     PASSED in 1.2s
//tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test      PASSED in 0.7s
//tensorflow/compiler/xla/service:dot_decomposer_test                    PASSED in 1.5s
//tensorflow/compiler/xla/service:dot_dimension_merger_test              PASSED in 1.4s
//tensorflow/compiler/xla/service:dot_merger_test                        PASSED in 1.3s
//tensorflow/compiler/xla/service:dynamic_dimension_inference_test       PASSED in 1.6s
//tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test      PASSED in 1.2s
//tensorflow/compiler/xla/service:dynamic_index_splitter_test            PASSED in 0.8s
//tensorflow/compiler/xla/service:dynamic_padder_test_cpu                PASSED in 16.5s
//tensorflow/compiler/xla/service:dynamic_parameter_binding_test         PASSED in 1.1s
//tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu          PASSED in 8.2s
//tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu          PASSED in 25.9s
//tensorflow/compiler/xla/service:flatten_call_graph_test                PASSED in 0.7s
//tensorflow/compiler/xla/service:float_normalization_test               PASSED in 0.8s
//tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test   PASSED in 1.3s
//tensorflow/compiler/xla/service:gather_expander_test                   PASSED in 1.2s
//tensorflow/compiler/xla/service:gather_simplifier_test                 PASSED in 1.3s
//tensorflow/compiler/xla/service:heap_simulator_test                    PASSED in 0.7s
//tensorflow/compiler/xla/service:hlo_alias_analysis_test                PASSED in 1.3s
//tensorflow/compiler/xla/service:hlo_casting_utils_test                 PASSED in 8.6s
//tensorflow/compiler/xla/service:hlo_computation_deduplicator_test      PASSED in 1.3s
//tensorflow/compiler/xla/service:hlo_computation_test                   PASSED in 4.0s
//tensorflow/compiler/xla/service:hlo_constant_folding_test              PASSED in 7.2s
//tensorflow/compiler/xla/service:hlo_cost_analysis_test                 PASSED in 9.0s
//tensorflow/compiler/xla/service:hlo_creation_utils_test                PASSED in 3.4s
//tensorflow/compiler/xla/service:hlo_cse_test                           PASSED in 7.6s
//tensorflow/compiler/xla/service:hlo_dataflow_analysis_test             PASSED in 2.2s
//tensorflow/compiler/xla/service:hlo_dce_test                           PASSED in 0.7s
//tensorflow/compiler/xla/service:hlo_domain_test                        PASSED in 2.5s
//tensorflow/compiler/xla/service:hlo_element_type_converter_test        PASSED in 0.9s
//tensorflow/compiler/xla/service:hlo_execution_profile_test             PASSED in 6.1s
//tensorflow/compiler/xla/service:hlo_graph_dumper_test                  PASSED in 0.7s
//tensorflow/compiler/xla/service:hlo_input_output_alias_config_test     PASSED in 0.8s
//tensorflow/compiler/xla/service:hlo_instruction_test                   PASSED in 0.6s
//tensorflow/compiler/xla/service:hlo_liveness_analysis_test             PASSED in 1.0s
//tensorflow/compiler/xla/service:hlo_memory_scheduler_test              PASSED in 2.4s
//tensorflow/compiler/xla/service:hlo_module_dce_test                    PASSED in 0.7s
//tensorflow/compiler/xla/service:hlo_module_metadata_test               PASSED in 0.2s
//tensorflow/compiler/xla/service:hlo_module_test                        PASSED in 0.8s
//tensorflow/compiler/xla/service:hlo_opcode_test                        PASSED in 0.2s
//tensorflow/compiler/xla/service:hlo_ordering_test                      PASSED in 0.8s
//tensorflow/compiler/xla/service:hlo_parser_test                        PASSED in 0.4s
//tensorflow/compiler/xla/service:hlo_pass_pipeline_test                 PASSED in 0.9s
//tensorflow/compiler/xla/service:hlo_phi_graph_test                     PASSED in 0.3s
//tensorflow/compiler/xla/service:hlo_proto_util_test                    PASSED in 0.8s
//tensorflow/compiler/xla/service:hlo_reachability_test                  PASSED in 0.6s
//tensorflow/compiler/xla/service:hlo_rematerialization_test             PASSED in 0.9s
//tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test  PASSED in 0.7s
//tensorflow/compiler/xla/service:hlo_replication_analysis_test          PASSED in 0.7s
//tensorflow/compiler/xla/service:hlo_schedule_test                      PASSED in 0.7s
//tensorflow/compiler/xla/service:hlo_sharding_test                      PASSED in 1.0s
//tensorflow/compiler/xla/service:hlo_value_semantics_analysis_test      PASSED in 1.8s
//tensorflow/compiler/xla/service:hlo_verifier_test                      PASSED in 1.1s
//tensorflow/compiler/xla/service:indexed_array_analysis_test            PASSED in 0.7s
//tensorflow/compiler/xla/service:instruction_fusion_test                PASSED in 1.4s
//tensorflow/compiler/xla/service:latency_hiding_scheduler_preparation_test PASSED in 1.1s
//tensorflow/compiler/xla/service:latency_hiding_scheduler_test          PASSED in 1.2s
//tensorflow/compiler/xla/service:layout_assignment_test                 PASSED in 6.0s
//tensorflow/compiler/xla/service:layout_normalization_test              PASSED in 4.0s
//tensorflow/compiler/xla/service:logistic_expander_test                 PASSED in 0.8s
//tensorflow/compiler/xla/service:loop_schedule_linearizer_test          PASSED in 0.8s
//tensorflow/compiler/xla/service:map_inliner_test                       PASSED in 0.6s
//tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test       PASSED in 0.1s
//tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.4s
//tensorflow/compiler/xla/service:memory_space_assignment_test           PASSED in 2.0s
//tensorflow/compiler/xla/service:memory_space_propagation_test          PASSED in 0.9s
//tensorflow/compiler/xla/service:name_uniquer_test                      PASSED in 0.3s
//tensorflow/compiler/xla/service:operand_upcaster_test                  PASSED in 0.8s
//tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 1.3s
//tensorflow/compiler/xla/service:pattern_matcher_gmock_test             PASSED in 0.7s
//tensorflow/compiler/xla/service:pattern_matcher_test                   PASSED in 0.7s
//tensorflow/compiler/xla/service:profile_guided_latency_estimator_test  PASSED in 1.7s
//tensorflow/compiler/xla/service:real_imag_expander_test                PASSED in 1.0s
//tensorflow/compiler/xla/service:reduce_decomposer_test                 PASSED in 0.8s
//tensorflow/compiler/xla/service:reduce_scatter_combiner_test           PASSED in 1.0s
//tensorflow/compiler/xla/service:reduce_scatter_decomposer_test         PASSED in 1.6s
//tensorflow/compiler/xla/service:reduce_scatter_reassociate_test        PASSED in 0.8s
//tensorflow/compiler/xla/service:reshape_decomposer_test                PASSED in 1.0s
//tensorflow/compiler/xla/service:reshape_mover_test                     PASSED in 1.0s
//tensorflow/compiler/xla/service:result_caster_test                     PASSED in 0.9s
//tensorflow/compiler/xla/service:root_instruction_sinker_test           PASSED in 0.7s
//tensorflow/compiler/xla/service:scatter_expander_test                  PASSED in 1.3s
//tensorflow/compiler/xla/service:scatter_simplifier_test                PASSED in 0.8s
//tensorflow/compiler/xla/service:select_and_scatter_expander_test       PASSED in 0.9s
//tensorflow/compiler/xla/service:shape_inference_test                   PASSED in 0.2s
//tensorflow/compiler/xla/service:shaped_buffer_test                     PASSED in 15.8s
//tensorflow/compiler/xla/service:sharding_propagation_test              PASSED in 5.5s
//tensorflow/compiler/xla/service:sharding_remover_test                  PASSED in 0.7s
//tensorflow/compiler/xla/service:simplify_fp_conversions_test           PASSED in 1.0s
//tensorflow/compiler/xla/service:slice_sinker_test                      PASSED in 0.9s
//tensorflow/compiler/xla/service:sort_simplifier_test                   PASSED in 0.8s
//tensorflow/compiler/xla/service:space_to_batch_converter_test          PASSED in 0.7s
//tensorflow/compiler/xla/service:stable_sort_expander_test              PASSED in 0.8s
//tensorflow/compiler/xla/service:stochastic_convert_decomposer_test     PASSED in 21.1s
//tensorflow/compiler/xla/service:stream_pool_test                       PASSED in 0.3s
//tensorflow/compiler/xla/service:topk_rewriter_test                     PASSED in 3.8s
//tensorflow/compiler/xla/service:transpose_folding_test                 PASSED in 1.9s
//tensorflow/compiler/xla/service:tuple_points_to_analysis_test          PASSED in 1.2s
//tensorflow/compiler/xla/service:tuple_simplifier_test                  PASSED in 1.1s
//tensorflow/compiler/xla/service:tuple_util_test                        PASSED in 1.2s
//tensorflow/compiler/xla/service:value_range_test                       PASSED in 1.0s
//tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 1.0s
//tensorflow/compiler/xla/service:while_loop_analysis_test               PASSED in 1.0s
//tensorflow/compiler/xla/service:while_loop_concat_code_motion_test     PASSED in 0.9s
//tensorflow/compiler/xla/service:while_loop_constant_sinking_test       PASSED in 1.0s
//tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.0s
//tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test  PASSED in 1.0s
//tensorflow/compiler/xla/service:while_loop_simplifier_test             PASSED in 1.2s
//tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test   PASSED in 1.8s
//tensorflow/compiler/xla/service:while_util_test                        PASSED in 0.9s
//tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test     PASSED in 8.7s
//tensorflow/compiler/xla/service:xla_debug_info_manager_test            PASSED in 2.6s
//tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test        PASSED in 0.8s
//tensorflow/compiler/xla/service/cpu:conv_canonicalization_test         PASSED in 0.9s
//tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test    PASSED in 1.0s
//tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test        PASSED in 1.1s
//tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test         PASSED in 2.1s
//tensorflow/compiler/xla/service/cpu:ir_emission_utils_test             PASSED in 1.0s
//tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test      PASSED in 2.6s
//tensorflow/compiler/xla/service/cpu:runtime_fft_test                   PASSED in 0.1s
//tensorflow/compiler/xla/service/cpu:shape_partition_test               PASSED in 0.8s
//tensorflow/compiler/xla/service/cpu:xfeed_manager_test                 PASSED in 1.1s
//tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test          PASSED in 0.6s
//tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test           PASSED in 6.8s
//tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 9.1s
//tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test  PASSED in 30.8s
//tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test              PASSED in 6.9s
//tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test              PASSED in 8.1s
//tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test           PASSED in 8.8s
//tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test      PASSED in 6.4s
//tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test     PASSED in 6.8s
//tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test             PASSED in 7.8s
//tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test             PASSED in 8.9s
//tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test           PASSED in 12.7s
//tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test        PASSED in 7.4s
//tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test                PASSED in 11.6s
//tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test       PASSED in 11.5s
//tensorflow/compiler/xla/service/cpu/tests:cpu_while_test               PASSED in 8.9s
//tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 8.0s
//tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test      PASSED in 1.1s
//tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test        PASSED in 1.2s
//tensorflow/compiler/xla/service/gpu:autotuner_util_test                PASSED in 0.4s
//tensorflow/compiler/xla/service/gpu:backend_configs_test               PASSED in 0.7s
//tensorflow/compiler/xla/service/gpu:copy_fusion_test                   PASSED in 1.6s
//tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test          PASSED in 1.2s
//tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test    PASSED in 1.1s
//tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test        PASSED in 1.6s
//tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test           PASSED in 1.1s
//tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test  PASSED in 1.0s
//tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test          PASSED in 1.4s
//tensorflow/compiler/xla/service/gpu:gpu_async_collective_annotator_test PASSED in 0.7s
//tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 1.0s
//tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test             PASSED in 1.4s
//tensorflow/compiler/xla/service/gpu:gpu_convert_async_collectives_to_sync_test PASSED in 1.0s
//tensorflow/compiler/xla/service/gpu:gpu_fusible_test                   PASSED in 1.5s
//tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test         PASSED in 1.0s
//tensorflow/compiler/xla/service/gpu:gpu_performance_model_test         PASSED in 1.3s
//tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test   PASSED in 4.0s
//tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test        PASSED in 0.3s
//tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test              PASSED in 0.8s
//tensorflow/compiler/xla/service/gpu:instruction_fusion_test            PASSED in 1.4s
//tensorflow/compiler/xla/service/gpu:ir_emission_utils_test             PASSED in 1.9s
//tensorflow/compiler/xla/service/gpu:matmul_utils_test                  PASSED in 1.6s
//tensorflow/compiler/xla/service/gpu:move_copy_to_users_test            PASSED in 1.3s
//tensorflow/compiler/xla/service/gpu:multi_output_fusion_test           PASSED in 1.7s
//tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.7s
//tensorflow/compiler/xla/service/gpu:priority_fusion_test               PASSED in 1.4s
//tensorflow/compiler/xla/service/gpu:reduction_splitter_test            PASSED in 0.8s
//tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test      PASSED in 1.4s
//tensorflow/compiler/xla/service/gpu:softmax_rewriter_triton_test       PASSED in 2.1s
//tensorflow/compiler/xla/service/gpu:target_util_test                   PASSED in 0.7s
//tensorflow/compiler/xla/service/gpu:topk_splitter_test                 PASSED in 45.0s
//tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test          PASSED in 0.9s
//tensorflow/compiler/xla/service/gpu:while_transformer_test             PASSED in 1.5s
//tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test        PASSED in 0.3s
//tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 0.8s
//tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 1.1s
//tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 1.5s
//tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 2.2s
//tensorflow/compiler/xla/service/graphcycles:graphcycles_test           PASSED in 0.7s
//tensorflow/compiler/xla/service/graphcycles:ordered_set_test           PASSED in 0.1s
//tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test            PASSED in 5.3s
//tensorflow/compiler/xla/service/llvm_ir:ir_array_test                  PASSED in 0.6s
//tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 0.6s
//tensorflow/compiler/xla/service/spmd:collective_permute_motion_test    PASSED in 0.8s
//tensorflow/compiler/xla/service/spmd:partition_assignment_test         PASSED in 2.0s
//tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 1.0s
//tensorflow/compiler/xla/service/spmd:spmd_partitioner_test             PASSED in 2.5s
//tensorflow/compiler/xla/service/spmd:spmd_prepare_test                 PASSED in 0.8s
//tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 1.0s
//tensorflow/compiler/xla/stream_executor:dnn_test                       PASSED in 0.4s
//tensorflow/compiler/xla/stream_executor:stream_test                    PASSED in 0.8s
//tensorflow/compiler/xla/stream_executor/host:host_stream_test          PASSED in 0.2s
//tensorflow/compiler/xla/stream_executor/tpu:c_api_conversions_test     PASSED in 0.6s
//tensorflow/compiler/xla/tests:all_reduce_test_cpu                      PASSED in 7.1s
//tensorflow/compiler/xla/tests:axpy_simple_test_cpu                     PASSED in 8.1s
//tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu        PASSED in 7.6s
//tensorflow/compiler/xla/tests:binop_scaling_test_cpu                   PASSED in 5.4s
//tensorflow/compiler/xla/tests:bitcast_convert_test_cpu                 PASSED in 6.7s
//tensorflow/compiler/xla/tests:broadcast_simple_test_cpu                PASSED in 9.7s
//tensorflow/compiler/xla/tests:broadcast_test_cpu                       PASSED in 12.0s
//tensorflow/compiler/xla/tests:buffer_donation_test_cpu                 PASSED in 7.8s
//tensorflow/compiler/xla/tests:call_test_cpu                            PASSED in 5.5s
//tensorflow/compiler/xla/tests:check_execution_arity_test_cpu           PASSED in 8.5s
//tensorflow/compiler/xla/tests:cholesky_test_cpu                        PASSED in 16.8s
//tensorflow/compiler/xla/tests:client_test_cpu                          PASSED in 7.8s
//tensorflow/compiler/xla/tests:collective_ops_test_cpu                  PASSED in 15.8s
//tensorflow/compiler/xla/tests:collective_pipeliner_execution_test_cpu  PASSED in 12.8s
//tensorflow/compiler/xla/tests:compilation_cache_test_cpu               PASSED in 5.9s
//tensorflow/compiler/xla/tests:compute_constant_test_cpu                PASSED in 7.3s
//tensorflow/compiler/xla/tests:concat_test_cpu                          PASSED in 11.1s
//tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu     PASSED in 10.3s
//tensorflow/compiler/xla/tests:constants_test_cpu                       PASSED in 8.0s
//tensorflow/compiler/xla/tests:convert_test_cpu                         PASSED in 15.0s
//tensorflow/compiler/xla/tests:copy_test_cpu                            PASSED in 10.8s
//tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu                  PASSED in 34.8s
//tensorflow/compiler/xla/tests:custom_call_test_cpu                     PASSED in 8.3s
//tensorflow/compiler/xla/tests:deallocation_test_cpu                    PASSED in 8.7s
//tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu               PASSED in 8.5s
//tensorflow/compiler/xla/tests:deep_graph_test_cpu                      PASSED in 8.2s
//tensorflow/compiler/xla/tests:fft_test_cpu                             PASSED in 8.3s
//tensorflow/compiler/xla/tests:float8_test_cpu                          PASSED in 9.4s
//tensorflow/compiler/xla/tests:floor_ceil_test_cpu                      PASSED in 9.6s
//tensorflow/compiler/xla/tests:fmax_fmin_test_cpu                       PASSED in 11.0s
//tensorflow/compiler/xla/tests:gather_operation_test_cpu                PASSED in 7.9s
//tensorflow/compiler/xla/tests:get_dimension_size_test_cpu              PASSED in 9.6s
//tensorflow/compiler/xla/tests:half_test_cpu                            PASSED in 8.8s
//tensorflow/compiler/xla/tests:hlo_metadata_test                        PASSED in 7.2s
//tensorflow/compiler/xla/tests:literal_test_util_test                   PASSED in 3.4s
//tensorflow/compiler/xla/tests:local_client_allocation_test_cpu         PASSED in 7.1s
//tensorflow/compiler/xla/tests:local_client_aot_test                    PASSED in 0.1s
//tensorflow/compiler/xla/tests:log_test_cpu                             PASSED in 8.0s
//tensorflow/compiler/xla/tests:map_test_cpu                             PASSED in 9.0s
//tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu               PASSED in 14.4s
//tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu          PASSED in 8.1s
//tensorflow/compiler/xla/tests:multiple_devices_on_host_test            PASSED in 6.7s
//tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu       PASSED in 8.8s
//tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu   PASSED in 8.3s
//tensorflow/compiler/xla/tests:pad_test_cpu                             PASSED in 7.0s
//tensorflow/compiler/xla/tests:pred_test_cpu                            PASSED in 8.2s
//tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu            PASSED in 9.4s
//tensorflow/compiler/xla/tests:reduce_hlo_test_cpu                      PASSED in 7.2s
//tensorflow/compiler/xla/tests:reduce_precision_test_cpu                PASSED in 10.1s
//tensorflow/compiler/xla/tests:replay_test_cpu                          PASSED in 8.6s
//tensorflow/compiler/xla/tests:reshape_motion_test_cpu                  PASSED in 7.5s
//tensorflow/compiler/xla/tests:reverse_test_cpu                         PASSED in 8.4s
//tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu       PASSED in 6.8s
//tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu             PASSED in 9.8s
//tensorflow/compiler/xla/tests:sample_text_test_cpu                     PASSED in 9.7s
//tensorflow/compiler/xla/tests:scatter_test_cpu                         PASSED in 12.8s
//tensorflow/compiler/xla/tests:select_test_cpu                          PASSED in 10.6s
//tensorflow/compiler/xla/tests:test_utils_test_cpu                      PASSED in 10.5s
//tensorflow/compiler/xla/tests:tile_assignment_test                     PASSED in 0.1s
//tensorflow/compiler/xla/tests:token_hlo_test_cpu                       PASSED in 9.5s
//tensorflow/compiler/xla/tests:topk_test_cpu                            PASSED in 8.2s
//tensorflow/compiler/xla/tests:transfer_manager_test_cpu                PASSED in 13.7s
//tensorflow/compiler/xla/tests:transpose_test_cpu                       PASSED in 10.1s
//tensorflow/compiler/xla/tests:tuple_test_cpu                           PASSED in 9.2s
//tensorflow/compiler/xla/tests:unary_op_test_cpu                        PASSED in 14.7s
//tensorflow/compiler/xla/tests:value_inference_test_cpu                 PASSED in 8.5s
//tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu               PASSED in 9.9s
//tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu               PASSED in 9.5s
//tensorflow/compiler/xla/tests:while_test_cpu                           PASSED in 9.9s
//tensorflow/compiler/xla/tests/fuzz:rand_000000_cpu                     PASSED in 7.4s
//tensorflow/compiler/xla/tests/fuzz:rand_000003_cpu                     PASSED in 8.4s
//tensorflow/compiler/xla/tests/fuzz:rand_000005_cpu                     PASSED in 7.6s
//tensorflow/compiler/xla/tests/fuzz:rand_000006_cpu                     PASSED in 7.2s
//tensorflow/compiler/xla/tests/fuzz:rand_000007_cpu                     PASSED in 5.4s
//tensorflow/compiler/xla/tests/fuzz:rand_000008_cpu                     PASSED in 7.9s
//tensorflow/compiler/xla/tests/fuzz:rand_000009_cpu                     PASSED in 8.7s
//tensorflow/compiler/xla/tests/fuzz:rand_000013_cpu                     PASSED in 10.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000015_cpu                     PASSED in 8.2s
//tensorflow/compiler/xla/tests/fuzz:rand_000016_cpu                     PASSED in 9.1s
//tensorflow/compiler/xla/tests/fuzz:rand_000017_cpu                     PASSED in 8.8s
//tensorflow/compiler/xla/tests/fuzz:rand_000018_cpu                     PASSED in 11.5s
//tensorflow/compiler/xla/tests/fuzz:rand_000019_cpu                     PASSED in 10.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000020_cpu                     PASSED in 6.5s
//tensorflow/compiler/xla/tests/fuzz:rand_000022_cpu                     PASSED in 16.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000024_cpu                     PASSED in 8.5s
//tensorflow/compiler/xla/tests/fuzz:rand_000025_cpu                     PASSED in 7.5s
//tensorflow/compiler/xla/tests/fuzz:rand_000026_cpu                     PASSED in 8.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000030_cpu                     PASSED in 6.8s
//tensorflow/compiler/xla/tests/fuzz:rand_000031_cpu                     PASSED in 7.2s
//tensorflow/compiler/xla/tests/fuzz:rand_000032_cpu                     PASSED in 8.2s
//tensorflow/compiler/xla/tests/fuzz:rand_000033_cpu                     PASSED in 8.5s
//tensorflow/compiler/xla/tests/fuzz:rand_000034_cpu                     PASSED in 8.8s
//tensorflow/compiler/xla/tests/fuzz:rand_000035_cpu                     PASSED in 8.7s
//tensorflow/compiler/xla/tests/fuzz:rand_000036_cpu                     PASSED in 7.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000039_cpu                     PASSED in 8.7s
//tensorflow/compiler/xla/tests/fuzz:rand_000040_cpu                     PASSED in 7.7s
//tensorflow/compiler/xla/tests/fuzz:rand_000041_cpu                     PASSED in 7.8s
//tensorflow/compiler/xla/tests/fuzz:rand_000043_cpu                     PASSED in 8.9s
//tensorflow/compiler/xla/tests/fuzz:rand_000049_cpu                     PASSED in 7.1s
//tensorflow/compiler/xla/tests/fuzz:rand_000053_cpu                     PASSED in 8.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000056_cpu                     PASSED in 7.1s
//tensorflow/compiler/xla/tests/fuzz:rand_000059_cpu                     PASSED in 7.3s
//tensorflow/compiler/xla/tests/fuzz:rand_000061_cpu                     PASSED in 7.5s
//tensorflow/compiler/xla/tests/fuzz:rand_000062_cpu                     PASSED in 10.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000064_cpu                     PASSED in 15.8s
//tensorflow/compiler/xla/tests/fuzz:rand_000066_cpu                     PASSED in 6.2s
//tensorflow/compiler/xla/tests/fuzz:rand_000069_cpu                     PASSED in 9.3s
//tensorflow/compiler/xla/tests/fuzz:rand_000071_cpu                     PASSED in 8.7s
//tensorflow/compiler/xla/tests/fuzz:rand_000077_cpu                     PASSED in 7.0s
//tensorflow/compiler/xla/tests/fuzz:rand_000078_cpu                     PASSED in 7.8s
//tensorflow/compiler/xla/tests/fuzz:rand_000079_cpu                     PASSED in 9.4s
//tensorflow/compiler/xla/tests/fuzz:rand_000081_cpu                     PASSED in 7.1s
//tensorflow/compiler/xla/tests/fuzz:rand_000084_cpu                     PASSED in 7.3s
//tensorflow/compiler/xla/tests/fuzz:rand_000085_cpu                     PASSED in 6.9s
//tensorflow/compiler/xla/tests/fuzz:rand_000086_cpu                     PASSED in 8.6s
//tensorflow/compiler/xla/tests/fuzz:rand_000088_cpu                     PASSED in 6.4s
//tensorflow/compiler/xla/tests/fuzz:rand_000089_cpu                     PASSED in 12.8s
//tensorflow/compiler/xla/tests/fuzz:rand_000090_cpu                     PASSED in 8.3s
//tensorflow/compiler/xla/tests/fuzz:rand_000092_cpu                     PASSED in 9.6s
//tensorflow/compiler/xla/tests/fuzz:rand_000094_cpu                     PASSED in 9.5s
//tensorflow/compiler/xla/tests/fuzz:rand_000095_cpu                     PASSED in 6.6s
//tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test         PASSED in 0.8s
//tensorflow/compiler/xla/tools:hlo_extractor_test                       PASSED in 0.7s
//tensorflow/compiler/xla/tools:hlo_module_loader_test                   PASSED in 0.7s
//tensorflow/compiler/xla/tools:hlo_slicer_test                          PASSED in 1.2s
//tensorflow/compiler/xla/tools:interactive_graphviz_bin_test            PASSED in 0.7s
//tensorflow/compiler/xla/tools:run_hlo_module_bin_test                  PASSED in 0.5s
//tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test         PASSED in 1.1s
//tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test           PASSED in 0.9s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test PASSED in 15.8s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test PASSED in 0.8s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test PASSED in 0.7s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test PASSED in 0.5s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test PASSED in 2.8s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test PASSED in 0.7s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test PASSED in 2.1s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test PASSED in 0.9s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:send_recv.hlotxt.test PASSED in 0.7s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test    PASSED in 0.5s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test PASSED in 0.7s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:stacktrace_to_location.hlo.test PASSED in 0.9s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test  PASSED in 0.6s
//tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test  PASSED in 1.9s
//tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test       PASSED in 0.9s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test      PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test     PASSED in 0.7s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test  PASSED in 2.2s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test   PASSED in 2.4s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test   PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test       PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test PASSED in 0.8s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test PASSED in 15.3s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test PASSED in 1.9s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_stacktrace.mlir.test PASSED in 0.9s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test PASSED in 1.3s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test PASSED in 1.2s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test PASSED in 0.7s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test PASSED in 0.5s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test   PASSED in 0.7s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test    PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test PASSED in 2.2s
//tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test PASSED in 0.6s
//tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test PASSED in 3.7s
//tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test PASSED in 0.8s
//tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests   PASSED in 8.1s
//tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests         PASSED in 0.8s
//tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test      PASSED in 37.9s
//tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.1s
//tensorflow/core:__tensorflow_core_lib_monitoring_counter_test          PASSED in 0.1s
//tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test            PASSED in 0.1s
//tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test       PASSED in 0.1s
//tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.1s
//tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test          PASSED in 0.2s
//tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test       PASSED in 0.2s
//tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.5s
//tensorflow/core:__tensorflow_core_lib_wav_wav_io_test                  PASSED in 0.3s
//tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs              PASSED in 0.1s
//tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests    PASSED in 0.4s
//tensorflow/core:lib_strings_ordered_code_test                          PASSED in 1.2s
//tensorflow/core:lib_strings_proto_serialization_test                   PASSED in 0.5s
//tensorflow/core/api_def:api_test                                       PASSED in 4.5s
//tensorflow/core/api_def:update_api_def_test                            PASSED in 0.2s
//tensorflow/core/common_runtime:all_to_all_test_cpu                     PASSED in 0.8s
//tensorflow/core/common_runtime:arg_ret_placement_test                  PASSED in 1.0s
//tensorflow/core/common_runtime:buf_rendezvous_test                     PASSED in 0.7s
//tensorflow/core/common_runtime:collective_executor_mgr_test            PASSED in 1.0s
//tensorflow/core/common_runtime:collective_param_resolver_local_test    PASSED in 11.0s
//tensorflow/core/common_runtime:collective_rma_local_test               PASSED in 0.7s
//tensorflow/core/common_runtime:composite_device_test                   PASSED in 0.7s
//tensorflow/core/common_runtime:cost_measurement_registry_test          PASSED in 3.0s
//tensorflow/core/common_runtime:cost_util_test                          PASSED in 0.1s
//tensorflow/core/common_runtime:device_mgr_test                         PASSED in 1.2s
//tensorflow/core/common_runtime:device_propagation_test                 PASSED in 0.6s
//tensorflow/core/common_runtime:device_resolver_local_test              PASSED in 1.0s
//tensorflow/core/common_runtime:device_set_test                         PASSED in 0.8s
//tensorflow/core/common_runtime:direct_session_test_cpu                 PASSED in 2.4s
//tensorflow/core/common_runtime:direct_session_with_debug_test          PASSED in 3.3s
//tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.3s
//tensorflow/core/common_runtime:dynamic_device_mgr_test                 PASSED in 0.9s
//tensorflow/core/common_runtime:eval_const_tensor_test                  PASSED in 1.1s
//tensorflow/core/common_runtime:executor_test                           PASSED in 2.1s
//tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 1.4s
//tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 0.9s
//tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 1.1s
//tensorflow/core/common_runtime:function_optimization_registry_test     PASSED in 1.0s
//tensorflow/core/common_runtime:function_threadpool_test                PASSED in 1.3s
//tensorflow/core/common_runtime:graph_constructor_test                  PASSED in 3.0s
//tensorflow/core/common_runtime:graph_runner_test                       PASSED in 0.9s
//tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu  PASSED in 4.0s
//tensorflow/core/common_runtime:inline_function_utils_test              PASSED in 0.8s
//tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 2.7s
//tensorflow/core/common_runtime:int32_fulltype_test                     PASSED in 0.7s
//tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 0.9s
//tensorflow/core/common_runtime:lower_case_op_test                      PASSED in 3.6s
//tensorflow/core/common_runtime:lower_function_call_test                PASSED in 4.4s
//tensorflow/core/common_runtime:lower_functional_ops_test               PASSED in 4.6s
//tensorflow/core/common_runtime:lower_if_op_test                        PASSED in 3.6s
//tensorflow/core/common_runtime:lower_while_op_test                     PASSED in 2.4s
//tensorflow/core/common_runtime:mkl_cpu_allocator_test                  PASSED in 0.1s
//tensorflow/core/common_runtime:mkl_threadpool_device_test              PASSED in 0.1s
//tensorflow/core/common_runtime:no_op_cost_measurement_test             PASSED in 0.5s
//tensorflow/core/common_runtime:null_request_cost_accessor_test         PASSED in 0.1s
//tensorflow/core/common_runtime:optimization_registry_test              PASSED in 1.3s
//tensorflow/core/common_runtime:optimize_cross_host_control_deps_test   PASSED in 7.3s
//tensorflow/core/common_runtime:optimize_function_graph_utils_test      PASSED in 0.8s
//tensorflow/core/common_runtime:partitioning_utils_test                 PASSED in 0.6s
//tensorflow/core/common_runtime:pending_counts_test                     PASSED in 0.9s
//tensorflow/core/common_runtime:permuter_test_cpu                       PASSED in 5.2s
//tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 1.0s
//tensorflow/core/common_runtime:placer_test                             PASSED in 1.4s
//tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.7s
//tensorflow/core/common_runtime:process_util_test                       PASSED in 0.1s
//tensorflow/core/common_runtime:quantize_training_test                  PASSED in 3.5s
//tensorflow/core/common_runtime:rendezvous_util_test                    PASSED in 0.1s
//tensorflow/core/common_runtime:replicate_per_replica_nodes_test        PASSED in 0.5s
//tensorflow/core/common_runtime:request_cost_accessor_registry_test     PASSED in 2.6s
//tensorflow/core/common_runtime:request_cost_test                       PASSED in 0.2s
//tensorflow/core/common_runtime:ring_gatherer_test_cpu                  PASSED in 3.2s
//tensorflow/core/common_runtime:ring_reducer_test_cpu                   PASSED in 5.7s
//tensorflow/core/common_runtime:scoped_allocator_mgr_test               PASSED in 4.1s
//tensorflow/core/common_runtime:session_test                            PASSED in 0.7s
//tensorflow/core/common_runtime:shape_refiner_test                      PASSED in 0.9s
//tensorflow/core/common_runtime:single_threaded_executor_test           PASSED in 1.0s
//tensorflow/core/common_runtime:threadpool_device_test                  PASSED in 1.4s
//tensorflow/core/common_runtime:type_inference_test                     PASSED in 2.2s
//tensorflow/core/common_runtime/eager:attr_builder_test                 PASSED in 34.8s
//tensorflow/core/common_runtime/eager:context_test                      PASSED in 18.0s
//tensorflow/core/common_runtime/eager:custom_device_test                PASSED in 14.6s
//tensorflow/core/common_runtime/eager:eager_executor_test               PASSED in 14.8s
//tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test    PASSED in 1.0s
//tensorflow/core/common_runtime/eager:eager_operation_test              PASSED in 14.0s
//tensorflow/core/common_runtime/eager:execute_node_test                 PASSED in 13.3s
//tensorflow/core/common_runtime/eager:execute_test                      PASSED in 32.6s
//tensorflow/core/common_runtime/eager:kernel_and_device_test            PASSED in 1.6s
//tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test         PASSED in 17.0s
//tensorflow/core/common_runtime/eager:placement_test                    PASSED in 15.9s
//tensorflow/core/common_runtime/eager:placement_utils_test              PASSED in 14.1s
//tensorflow/core/common_runtime/eager:summary_optimizer_test            PASSED in 0.1s
//tensorflow/core/common_runtime/eager:tensor_handle_data_test           PASSED in 17.7s
//tensorflow/core/common_runtime/eager:tensor_handle_test                PASSED in 13.7s
//tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test  PASSED in 0.2s
//tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 33.8s
//tensorflow/core/common_runtime/next_pluggable_device/c:tf_rendezvous_c_api_conversions_test PASSED in 0.1s
//tensorflow/core/config:flags_py_test                                   PASSED in 9.9s
//tensorflow/core/config:flags_test                                      PASSED in 0.1s
//tensorflow/core/data:compression_utils_test                            PASSED in 2.3s
//tensorflow/core/data:dataset_utils_test                                PASSED in 2.2s
//tensorflow/core/data:hash_utils_test                                   PASSED in 1.0s
//tensorflow/core/data:metric_utils_test                                 PASSED in 6.1s
//tensorflow/core/data:name_utils_test                                   PASSED in 0.3s
//tensorflow/core/data:rewrite_utils_test                                PASSED in 1.0s
//tensorflow/core/data:serialization_utils_test                          PASSED in 1.0s
//tensorflow/core/data:snapshot_utils_test                               PASSED in 0.8s
//tensorflow/core/data:split_utils_test                                  PASSED in 0.5s
//tensorflow/core/data:standalone_save_restore_test                      PASSED in 2.4s
//tensorflow/core/data:standalone_test                                   PASSED in 5.3s
//tensorflow/core/data:tfdataz_metrics_test                              PASSED in 2.6s
//tensorflow/core/data:unbounded_thread_pool_test                        PASSED in 0.4s
//tensorflow/core/data/service:auto_scaler_test                          PASSED in 0.3s
//tensorflow/core/data/service:auto_shard_rewriter_test                  PASSED in 1.2s
//tensorflow/core/data/service:common_test                               PASSED in 0.1s
//tensorflow/core/data/service:credentials_factory_test                  PASSED in 4.1s
//tensorflow/core/data/service:cross_trainer_cache_test                  PASSED in 1.3s
//tensorflow/core/data/service:data_service_test                         PASSED in 12.9s
//tensorflow/core/data/service:data_transfer_test                        PASSED in 0.6s
//tensorflow/core/data/service:dataset_store_test                        PASSED in 0.8s
//tensorflow/core/data/service:dispatcher_client_test                    PASSED in 7.4s
//tensorflow/core/data/service:dispatcher_state_test                     PASSED in 0.8s
//tensorflow/core/data/service:grpc_dispatcher_impl_test                 PASSED in 3.8s
//tensorflow/core/data/service:grpc_util_test                            PASSED in 0.8s
//tensorflow/core/data/service:grpc_worker_impl_test                     PASSED in 3.9s
//tensorflow/core/data/service:journal_test                              PASSED in 0.6s
//tensorflow/core/data/service:logging_utils_test                        PASSED in 0.2s
//tensorflow/core/data/service:task_runner_test                          PASSED in 4.1s
//tensorflow/core/data/service:test_util_test                            PASSED in 3.4s
//tensorflow/core/data/service:url_test                                  PASSED in 0.3s
//tensorflow/core/data/service:utils_test                                PASSED in 0.8s
//tensorflow/core/data/service:validate_utils_test                       PASSED in 0.2s
//tensorflow/core/data/service:worker_client_test                        PASSED in 3.9s
//tensorflow/core/data/service:worker_impl_test                          PASSED in 3.3s
//tensorflow/core/data/service/client:data_service_client_test           PASSED in 4.6s
//tensorflow/core/data/service/client:utils_test                         PASSED in 3.7s
//tensorflow/core/data/service/client:validate_utils_test                PASSED in 2.3s
//tensorflow/core/data/service/snapshot:distributed_snapshot_test        PASSED in 20.5s
//tensorflow/core/data/service/snapshot:file_utils_test                  PASSED in 0.8s
//tensorflow/core/data/service/snapshot:path_utils_test                  PASSED in 0.1s
//tensorflow/core/data/service/snapshot:snapshot_manager_test            PASSED in 3.5s
//tensorflow/core/data/service/snapshot:snapshot_split_provider_test     PASSED in 0.8s
//tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 4.8s
//tensorflow/core/data/service/snapshot:snapshot_stream_writer_test      PASSED in 4.8s
//tensorflow/core/data/service/snapshot:utils_test                       PASSED in 0.1s
//tensorflow/core/debug:debug_graph_utils_test                           PASSED in 0.6s
//tensorflow/core/distributed_runtime:call_options_test                  PASSED in 0.7s
//tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 4.9s
//tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 1.0s
//tensorflow/core/distributed_runtime:collective_rma_distributed_test    PASSED in 0.9s
//tensorflow/core/distributed_runtime:device_resolver_distributed_test   PASSED in 1.1s
//tensorflow/core/distributed_runtime:message_wrappers_test              PASSED in 0.1s
//tensorflow/core/distributed_runtime:partial_run_mgr_test               PASSED in 1.7s
//tensorflow/core/distributed_runtime:recent_request_ids_test            PASSED in 0.1s
//tensorflow/core/distributed_runtime:request_id_test                    PASSED in 0.3s
//tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test   PASSED in 1.4s
//tensorflow/core/distributed_runtime:server_lib_test                    PASSED in 0.1s
//tensorflow/core/distributed_runtime:session_mgr_test                   PASSED in 0.8s
//tensorflow/core/distributed_runtime:tensor_coding_test                 PASSED in 0.4s
//tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.7s
//tensorflow/core/distributed_runtime/eager:eager_service_impl_test      PASSED in 24.5s
//tensorflow/core/distributed_runtime/eager:remote_mgr_test              PASSED in 9.9s
//tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 40.6s
//tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 48.5s
//tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 45.4s
//tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test        PASSED in 4.3s
//tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test         PASSED in 3.4s
//tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test   PASSED in 0.7s
//tensorflow/core/example:example_parser_configuration_test              PASSED in 2.0s
//tensorflow/core/example:feature_util_test                              PASSED in 0.3s
//tensorflow/core/framework:allocator_test                               PASSED in 3.4s
//tensorflow/core/framework:attr_value_util_test                         PASSED in 0.9s
//tensorflow/core/framework:batch_util_test                              PASSED in 2.2s
//tensorflow/core/framework:bfloat16_test                                PASSED in 1.5s
//tensorflow/core/framework:common_shape_fns_test                        PASSED in 0.9s
//tensorflow/core/framework:dataset_test                                 PASSED in 0.8s
//tensorflow/core/framework:device_base_test                             PASSED in 1.2s
//tensorflow/core/framework:disable_jit_test                             PASSED in 2.7s
//tensorflow/core/framework:framework_op_gen_lib_test                    PASSED in 0.1s
//tensorflow/core/framework:framework_op_segment_test                    PASSED in 1.1s
//tensorflow/core/framework:framework_resource_var_test                  PASSED in 0.1s
//tensorflow/core/framework:framework_run_handler_test                   PASSED in 2.1s
//tensorflow/core/framework:framework_run_handler_util_test              PASSED in 2.1s
//tensorflow/core/framework:full_type_inference_util_test                PASSED in 0.9s
//tensorflow/core/framework:full_type_util_test                          PASSED in 0.8s
//tensorflow/core/framework:function_test                                PASSED in 0.9s
//tensorflow/core/framework:graph_def_util_test                          PASSED in 0.7s
//tensorflow/core/framework:graph_to_functiondef_test                    PASSED in 0.8s
//tensorflow/core/framework:kernel_def_builder_test                      PASSED in 1.0s
//tensorflow/core/framework:kernel_def_util_test                         PASSED in 1.0s
//tensorflow/core/framework:memory_types_test                            PASSED in 0.9s
//tensorflow/core/framework:model_test                                   PASSED in 1.7s
//tensorflow/core/framework:node_def_builder_test                        PASSED in 0.8s
//tensorflow/core/framework:node_def_util_test                           PASSED in 0.8s
//tensorflow/core/framework:node_properties_test                         PASSED in 0.8s
//tensorflow/core/framework:op_compatibility_test                        PASSED in 1.2s
//tensorflow/core/framework:op_def_builder_test                          PASSED in 1.0s
//tensorflow/core/framework:op_def_util_test                             PASSED in 0.7s
//tensorflow/core/framework:op_kernel_test                               PASSED in 1.5s
//tensorflow/core/framework:op_registration_test                         PASSED in 1.0s
//tensorflow/core/framework:partial_tensor_shape_test                    PASSED in 0.7s
//tensorflow/core/framework:rendezvous_test                              PASSED in 3.5s
//tensorflow/core/framework:resource_handle_test                         PASSED in 0.3s
//tensorflow/core/framework:resource_mgr_test                            PASSED in 1.9s
//tensorflow/core/framework:resource_op_kernel_test                      PASSED in 1.0s
//tensorflow/core/framework:shape_inference_test                         PASSED in 0.8s
//tensorflow/core/framework:shape_inference_testutil_test                PASSED in 1.8s
//tensorflow/core/framework:tensor_shape_test                            PASSED in 7.8s
//tensorflow/core/framework:tensor_slice_test                            PASSED in 1.1s
//tensorflow/core/framework:tensor_test                                  PASSED in 37.5s
//tensorflow/core/framework:tensor_testutil_test                         PASSED in 0.8s
//tensorflow/core/framework:tensor_util_test                             PASSED in 1.2s
//tensorflow/core/framework:tracking_allocator_test                      PASSED in 0.9s
//tensorflow/core/framework:types_test                                   PASSED in 1.2s
//tensorflow/core/framework:variant_op_registry_test                     PASSED in 17.7s
//tensorflow/core/framework:variant_test                                 PASSED in 1.1s
//tensorflow/core/framework/registration:registration_test               PASSED in 0.5s
//tensorflow/core/function/capture:by_ref_capture_test                   PASSED in 11.5s
//tensorflow/core/function/capture:capture_container_test                PASSED in 9.5s
//tensorflow/core/function/integration_test:side_inputs_manual_api_test  PASSED in 43.7s
//tensorflow/core/function/integration_test:side_inputs_test             PASSED in 22.3s
//tensorflow/core/function/polymorphism:function_cache_test              PASSED in 15.8s
//tensorflow/core/function/polymorphism:function_type_test               PASSED in 15.5s
//tensorflow/core/function/polymorphism:type_dispatch_test               PASSED in 10.7s
//tensorflow/core/function/runtime_client:runtime_client_cc_test         PASSED in 50.2s
//tensorflow/core/function/trace_type:default_types_test                 PASSED in 12.1s
//tensorflow/core/function/trace_type:serialization_test                 PASSED in 9.3s
//tensorflow/core/function/trace_type:trace_type_test                    PASSED in 15.9s
//tensorflow/core/graph:algorithm_test                                   PASSED in 1.5s
//tensorflow/core/graph:collective_order_test                            PASSED in 0.5s
//tensorflow/core/graph:control_flow_test                                PASSED in 1.5s
//tensorflow/core/graph:costmodel_test                                   PASSED in 1.4s
//tensorflow/core/graph:edgeset_test                                     PASSED in 0.9s
//tensorflow/core/graph:graph_debug_info_builder_test                    PASSED in 2.0s
//tensorflow/core/graph:graph_def_builder_test                           PASSED in 1.3s
//tensorflow/core/graph:graph_partition_test                             PASSED in 0.9s
//tensorflow/core/graph:graph_test                                       PASSED in 0.9s
//tensorflow/core/graph:node_builder_test                                PASSED in 1.3s
//tensorflow/core/graph:optimizer_cse_test                               PASSED in 0.7s
//tensorflow/core/graph:subgraph_test                                    PASSED in 1.0s
//tensorflow/core/graph:tensor_id_test                                   PASSED in 1.0s
//tensorflow/core/graph:validate_test                                    PASSED in 1.0s
//tensorflow/core/graph/regularization:simple_delete_test                PASSED in 0.2s
//tensorflow/core/graph/regularization:util_test                         PASSED in 0.2s
//tensorflow/core/grappler:graph_topology_view_test                      PASSED in 0.1s
//tensorflow/core/grappler:graph_view_test                               PASSED in 2.5s
//tensorflow/core/grappler:grappler_item_builder_test                    PASSED in 2.3s
//tensorflow/core/grappler:grappler_item_test                            PASSED in 2.3s
//tensorflow/core/grappler:mutable_graph_view_test                       PASSED in 1.8s
//tensorflow/core/grappler:utils_test                                    PASSED in 4.2s
//tensorflow/core/grappler/clusters:single_machine_test                  PASSED in 25.3s
//tensorflow/core/grappler/clusters:virtual_cluster_test                 PASSED in 1.7s
//tensorflow/core/grappler/costs:analytical_cost_estimator_test          PASSED in 1.9s
//tensorflow/core/grappler/costs:cost_estimator_test                     PASSED in 0.2s
//tensorflow/core/grappler/costs:graph_memory_test                       PASSED in 2.2s
//tensorflow/core/grappler/costs:graph_properties_test                   PASSED in 4.3s
//tensorflow/core/grappler/costs:robust_stats_test                       PASSED in 0.1s
//tensorflow/core/grappler/costs:utils_test                              PASSED in 2.9s
//tensorflow/core/grappler/costs:virtual_placer_test                     PASSED in 0.4s
//tensorflow/core/grappler/costs:virtual_scheduler_test                  PASSED in 2.5s
//tensorflow/core/grappler/graph_analyzer:gen_node_test                  PASSED in 2.0s
//tensorflow/core/grappler/graph_analyzer:graph_analyzer_test            PASSED in 2.6s
//tensorflow/core/grappler/graph_analyzer:hash_tools_test                PASSED in 2.0s
//tensorflow/core/grappler/graph_analyzer:sig_node_test                  PASSED in 5.5s
//tensorflow/core/grappler/graph_analyzer:subgraph_test                  PASSED in 2.4s
//tensorflow/core/grappler/inputs:utils_test                             PASSED in 0.2s
//tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu      PASSED in 7.1s
//tensorflow/core/grappler/optimizers:auto_parallel_test_cpu             PASSED in 3.7s
//tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 2.8s
//tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 3.7s
//tensorflow/core/grappler/optimizers:debug_stripper_test_cpu            PASSED in 2.6s
//tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu      PASSED in 2.1s
//tensorflow/core/grappler/optimizers:evaluation_utils_test              PASSED in 0.8s
//tensorflow/core/grappler/optimizers:function_api_info_test             PASSED in 0.1s
//tensorflow/core/grappler/optimizers:function_optimizer_test_cpu        PASSED in 4.5s
//tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu  PASSED in 3.7s
//tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.2s
//tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.0s
//tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu     PASSED in 1.9s
//tensorflow/core/grappler/optimizers:implementation_selector_test       PASSED in 3.6s
//tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu            PASSED in 2.8s
//tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu          PASSED in 3.2s
//tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu            PASSED in 12.0s
//tensorflow/core/grappler/optimizers:mkl_remapper_test                  PASSED in 2.4s
//tensorflow/core/grappler/optimizers:model_pruner_test_cpu              PASSED in 2.7s
//tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu     PASSED in 3.7s
//tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test    PASSED in 4.9s
//tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu           PASSED in 2.7s
//tensorflow/core/grappler/optimizers:static_schedule_test_cpu           PASSED in 2.1s
//tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test            PASSED in 0.8s
//tensorflow/core/grappler/optimizers/data:auto_shard_test               PASSED in 0.7s
//tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test    PASSED in 1.1s
//tensorflow/core/grappler/optimizers/data:batch_parallelization_test    PASSED in 0.5s
//tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 1.8s
//tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.5s
//tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test  PASSED in 0.9s
//tensorflow/core/grappler/optimizers/data:filter_fusion_test            PASSED in 0.5s
//tensorflow/core/grappler/optimizers/data:filter_parallelization_test   PASSED in 0.6s
//tensorflow/core/grappler/optimizers/data:function_utils_test           PASSED in 1.0s
//tensorflow/core/grappler/optimizers/data:fusion_utils_test             PASSED in 0.7s
//tensorflow/core/grappler/optimizers/data:graph_utils_test              PASSED in 0.6s
//tensorflow/core/grappler/optimizers/data:inject_prefetch_test          PASSED in 0.5s
//tensorflow/core/grappler/optimizers/data:make_deterministic_test       PASSED in 0.6s
//tensorflow/core/grappler/optimizers/data:make_sloppy_test              PASSED in 0.6s
//tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test     PASSED in 0.4s
//tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test    PASSED in 0.6s
//tensorflow/core/grappler/optimizers/data:map_fusion_test               PASSED in 0.6s
//tensorflow/core/grappler/optimizers/data:map_parallelization_test      PASSED in 0.5s
//tensorflow/core/grappler/optimizers/data:noop_elimination_test         PASSED in 0.4s
//tensorflow/core/grappler/optimizers/data:parallel_batch_test           PASSED in 0.4s
//tensorflow/core/grappler/optimizers/data:replicate_on_split_test       PASSED in 0.5s
//tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.4s
//tensorflow/core/grappler/optimizers/data:slack_test                    PASSED in 0.8s
//tensorflow/core/grappler/optimizers/data:split_utils_test              PASSED in 1.7s
//tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test  PASSED in 0.6s
//tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test   PASSED in 0.1s
//tensorflow/core/grappler/utils:canonicalizer_test                      PASSED in 1.5s
//tensorflow/core/grappler/utils:colocation_test                         PASSED in 0.5s
//tensorflow/core/grappler/utils:frame_test                              PASSED in 0.1s
//tensorflow/core/grappler/utils:functions_test                          PASSED in 2.2s
//tensorflow/core/grappler/utils:graph_view_internal_test                PASSED in 0.7s
//tensorflow/core/grappler/utils:graph_view_test                         PASSED in 2.1s
//tensorflow/core/grappler/utils:grappler_test_test                      PASSED in 6.5s
//tensorflow/core/grappler/utils:pattern_utils_test                      PASSED in 0.6s
//tensorflow/core/grappler/utils:scc_test                                PASSED in 1.4s
//tensorflow/core/grappler/utils:symbolic_shapes_test                    PASSED in 1.1s
//tensorflow/core/grappler/utils:topological_sort_test                   PASSED in 0.6s
//tensorflow/core/grappler/utils:tpu_test                                PASSED in 0.1s
//tensorflow/core/grappler/utils:transitive_fanin_test                   PASSED in 0.5s
//tensorflow/core/grappler/utils:traversal_test                          PASSED in 0.6s
//tensorflow/core/grappler/verifiers:structure_verifier_test             PASSED in 4.1s
//tensorflow/core/ir:interfaces_test                                     PASSED in 0.3s
//tensorflow/core/ir:ops_test                                            PASSED in 0.5s
//tensorflow/core/ir:shape_inference_utils_test                          PASSED in 0.3s
//tensorflow/core/ir:tf_op_registry_test                                 PASSED in 0.2s
//tensorflow/core/ir:tf_op_wrapper_test                                  PASSED in 0.3s
//tensorflow/core/ir:utility_test                                        PASSED in 0.2s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 11.3s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 1.2s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 1.0s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 1.0s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 0.8s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.8s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.8s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.9s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 10.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.5s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 0.9s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.8s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.9s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 1.0s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 1.2s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 0.8s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 10.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 1.2s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 11.3s
//tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 1.2s
//tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 1.2s
//tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.6s
//tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 1.0s
//tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 0.7s
//tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 0.5s
//tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.4s
//tensorflow/core/ir/tests:attributes.mlir.test                          PASSED in 0.5s
//tensorflow/core/ir/tests:canonicalize.mlir.test                        PASSED in 0.7s
//tensorflow/core/ir/tests:compatible_types.mlir.test                    PASSED in 0.8s
//tensorflow/core/ir/tests:concrete-ops.mlir.test                        PASSED in 0.7s
//tensorflow/core/ir/tests:generic_concrete_ops.mlir.test                PASSED in 0.7s
//tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test                PASSED in 0.9s
//tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test             PASSED in 0.9s
//tensorflow/core/ir/tests:invalid.mlir.test                             PASSED in 0.6s
//tensorflow/core/ir/tests:invalid_types.mlir.test                       PASSED in 0.9s
//tensorflow/core/ir/tests:ops.mlir.test                                 PASSED in 1.1s
//tensorflow/core/ir/tests:region-invalid-ops.mlir.test                  PASSED in 10.6s
//tensorflow/core/ir/tests:region-ops-graph.mlir.test                    PASSED in 0.8s
//tensorflow/core/ir/tests:region-ops.mlir.test                          PASSED in 10.6s
//tensorflow/core/ir/tests:types.mlir.test                               PASSED in 10.6s
//tensorflow/core/ir/types:dialect_test                                  PASSED in 0.1s
//tensorflow/core/kernels:as_string_op_test                              PASSED in 0.5s
//tensorflow/core/kernels:basic_ops_benchmark_test                       PASSED in 0.7s
//tensorflow/core/kernels:batch_kernels_env_test                         PASSED in 1.2s
//tensorflow/core/kernels:batch_kernels_test                             PASSED in 32.9s
//tensorflow/core/kernels:bias_op_test                                   PASSED in 0.6s
//tensorflow/core/kernels:bincount_op_test_cpu                           PASSED in 0.7s
//tensorflow/core/kernels:broadcast_to_op_test_cpu                       PASSED in 0.7s
//tensorflow/core/kernels:cast_op_test_cpu                               PASSED in 55.2s
//tensorflow/core/kernels:checkpoint_callback_manager_test               PASSED in 1.5s
//tensorflow/core/kernels:clustering_ops_test                            PASSED in 0.7s
//tensorflow/core/kernels:composite_tensor_variant_test                  PASSED in 1.2s
//tensorflow/core/kernels:concat_op_test                                 PASSED in 0.8s
//tensorflow/core/kernels:constant_op_test_cpu                           PASSED in 0.8s
//tensorflow/core/kernels:control_flow_ops_test                          PASSED in 8.7s
//tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu        PASSED in 0.6s
//tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu         PASSED in 0.6s
//tensorflow/core/kernels:conv_ops_benchmark_test_cpu                    PASSED in 0.7s
//tensorflow/core/kernels:conv_ops_test_cpu                              PASSED in 5.8s
//tensorflow/core/kernels:count_ops_test                                 PASSED in 0.9s
//tensorflow/core/kernels:cross_op_test                                  PASSED in 0.6s
//tensorflow/core/kernels:cwise_ops_test_cpu                             PASSED in 1.0s
//tensorflow/core/kernels:debug_ops_test                                 PASSED in 0.8s
//tensorflow/core/kernels:decode_wav_op_test                             PASSED in 2.1s
//tensorflow/core/kernels:deep_conv2d_test                               PASSED in 0.5s
//tensorflow/core/kernels:dequantize_op_test                             PASSED in 1.1s
//tensorflow/core/kernels:diag_op_test_cpu                               PASSED in 0.8s
//tensorflow/core/kernels:dynamic_partition_op_test_cpu                  PASSED in 0.6s
//tensorflow/core/kernels:dynamic_stitch_op_test_cpu                     PASSED in 0.6s
//tensorflow/core/kernels:eigen_activations_test                         PASSED in 0.5s
//tensorflow/core/kernels:eigen_attention_test                           PASSED in 0.1s
//tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test        PASSED in 0.9s
//tensorflow/core/kernels:eigen_backward_spatial_convolutions_test       PASSED in 0.5s
//tensorflow/core/kernels:eigen_benchmark_cpu_test                       PASSED in 0.1s
//tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test           PASSED in 0.1s
//tensorflow/core/kernels:eigen_pooling_test                             PASSED in 0.4s
//tensorflow/core/kernels:encode_wav_op_test                             PASSED in 1.8s
//tensorflow/core/kernels:fingerprint_op_test                            PASSED in 0.7s
//tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu                PASSED in 1.2s
//tensorflow/core/kernels:fused_batch_norm_op_test_cpu                   PASSED in 0.7s
//tensorflow/core/kernels:gather_nd_op_test_cpu                          PASSED in 1.3s
//tensorflow/core/kernels:gather_op_test_cpu                             PASSED in 0.5s
//tensorflow/core/kernels:guarantee_const_op_test                        PASSED in 0.7s
//tensorflow/core/kernels:identity_n_op_test                             PASSED in 0.6s
//tensorflow/core/kernels:identity_op_test                               PASSED in 0.5s
//tensorflow/core/kernels:immutable_constant_op_test                     PASSED in 0.7s
//tensorflow/core/kernels:in_topk_op_test                                PASSED in 0.6s
//tensorflow/core/kernels:isotonic_regression_op_test                    PASSED in 0.7s
//tensorflow/core/kernels:logging_ops_test                               PASSED in 1.9s
//tensorflow/core/kernels:lookup_ops_test                                PASSED in 0.8s
//tensorflow/core/kernels:loss_test                                      PASSED in 0.4s
//tensorflow/core/kernels:lrn_op_test_cpu                                PASSED in 0.5s
//tensorflow/core/kernels:matmul_op_test_cpu                             PASSED in 4.2s
//tensorflow/core/kernels:merge_v2_checkpoints_op_test                   PASSED in 0.7s
//tensorflow/core/kernels:mfcc_dct_test                                  PASSED in 0.1s
//tensorflow/core/kernels:mfcc_mel_filterbank_test                       PASSED in 1.2s
//tensorflow/core/kernels:mfcc_op_test_cpu                               PASSED in 4.9s
//tensorflow/core/kernels:mfcc_test                                      PASSED in 0.2s
//tensorflow/core/kernels:multinomial_op_test_cpu                        PASSED in 0.6s
//tensorflow/core/kernels:nn_ops_test_cpu                                PASSED in 0.8s
//tensorflow/core/kernels:one_hot_op_test                                PASSED in 0.7s
//tensorflow/core/kernels:ops_testutil_test                              PASSED in 0.7s
//tensorflow/core/kernels:ops_util_test                                  PASSED in 0.3s
//tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu     PASSED in 0.8s
//tensorflow/core/kernels:parse_tensor_test                              PASSED in 5.8s
//tensorflow/core/kernels:quantization_utils_test                        PASSED in 0.7s
//tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu            PASSED in 0.6s
//tensorflow/core/kernels:quantize_down_and_shrink_range_op_test         PASSED in 1.1s
//tensorflow/core/kernels:quantize_op_test                               PASSED in 0.7s
//tensorflow/core/kernels:quantized_activation_ops_test                  PASSED in 1.0s
//tensorflow/core/kernels:quantized_add_op_test                          PASSED in 1.7s
//tensorflow/core/kernels:quantized_batch_norm_op_test                   PASSED in 0.6s
//tensorflow/core/kernels:quantized_bias_add_op_test                     PASSED in 0.7s
//tensorflow/core/kernels:quantized_concat_op_test                       PASSED in 0.7s
//tensorflow/core/kernels:quantized_conv_ops_test                        PASSED in 0.7s
//tensorflow/core/kernels:quantized_instance_norm_test                   PASSED in 0.7s
//tensorflow/core/kernels:quantized_matmul_op_test                       PASSED in 1.0s
//tensorflow/core/kernels:quantized_mul_op_test                          PASSED in 2.0s
//tensorflow/core/kernels:quantized_pooling_ops_test                     PASSED in 0.6s
//tensorflow/core/kernels:quantized_reshape_op_test                      PASSED in 0.6s
//tensorflow/core/kernels:quantized_resize_bilinear_op_test              PASSED in 1.7s
//tensorflow/core/kernels:ragged_fill_empty_rows_op_test                 PASSED in 16.0s
//tensorflow/core/kernels:ragged_gather_op_test                          PASSED in 1.5s
//tensorflow/core/kernels:ragged_range_op_test                           PASSED in 1.0s
//tensorflow/core/kernels:ragged_tensor_from_variant_op_test             PASSED in 0.8s
//tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test            PASSED in 1.2s
//tensorflow/core/kernels:ragged_tensor_to_tensor_op_test                PASSED in 0.6s
//tensorflow/core/kernels:ragged_tensor_to_variant_op_test               PASSED in 1.5s
//tensorflow/core/kernels:random_binomial_op_test_cpu                    PASSED in 1.0s
//tensorflow/core/kernels:random_index_shuffle_test                      PASSED in 0.3s
//tensorflow/core/kernels:random_op_test_cpu                             PASSED in 0.6s
//tensorflow/core/kernels:random_poisson_op_test_cpu                     PASSED in 1.1s
//tensorflow/core/kernels:range_sampler_test                             PASSED in 0.4s
//tensorflow/core/kernels:reduction_ops_test_cpu                         PASSED in 1.0s
//tensorflow/core/kernels:regex_replace_op_test                          PASSED in 1.4s
//tensorflow/core/kernels:requantization_range_op_test                   PASSED in 0.7s
//tensorflow/core/kernels:requantize_op_test                             PASSED in 0.6s
//tensorflow/core/kernels:resource_ops_test                              PASSED in 1.3s
//tensorflow/core/kernels:restore_op_test                                PASSED in 0.5s
//tensorflow/core/kernels:restore_v2_op_test                             PASSED in 0.5s
//tensorflow/core/kernels:reverse_op_test                                PASSED in 0.8s
//tensorflow/core/kernels:roll_op_test                                   PASSED in 0.5s
//tensorflow/core/kernels:save_op_test                                   PASSED in 0.5s
//tensorflow/core/kernels:save_v2_op_test                                PASSED in 0.9s
//tensorflow/core/kernels:scan_ops_test_cpu                              PASSED in 0.6s
//tensorflow/core/kernels:scatter_nd_op_test_cpu                         PASSED in 0.6s
//tensorflow/core/kernels:scatter_op_test                                PASSED in 0.5s
//tensorflow/core/kernels:scoped_allocator_ops_test_cpu                  PASSED in 7.4s
//tensorflow/core/kernels:sdca_ops_test                                  PASSED in 1.7s
//tensorflow/core/kernels:segment_reduction_ops_test                     PASSED in 0.7s
//tensorflow/core/kernels:sendrecv_ops_test                              PASSED in 0.5s
//tensorflow/core/kernels:sequence_ops_test                              PASSED in 0.7s
//tensorflow/core/kernels:shape_ops_test                                 PASSED in 10.4s
//tensorflow/core/kernels:slice_op_test                                  PASSED in 0.6s
//tensorflow/core/kernels:spacetobatch_benchmark_test_cpu                PASSED in 1.8s
//tensorflow/core/kernels:sparse_add_op_test                             PASSED in 11.1s
//tensorflow/core/kernels:sparse_dense_binary_op_shared_test             PASSED in 0.6s
//tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu             PASSED in 0.5s
//tensorflow/core/kernels:sparse_matmul_op_test_cpu                      PASSED in 0.4s
//tensorflow/core/kernels:sparse_reduce_sum_op_test                      PASSED in 0.9s
//tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu         PASSED in 0.5s
//tensorflow/core/kernels:sparse_to_dense_op_test_cpu                    PASSED in 0.8s
//tensorflow/core/kernels:sparse_utils_test                              PASSED in 0.4s
//tensorflow/core/kernels:sparse_xent_op_test_cpu                        PASSED in 0.5s
//tensorflow/core/kernels:spectrogram_op_test_cpu                        PASSED in 2.3s
//tensorflow/core/kernels:spectrogram_test                               PASSED in 0.1s
//tensorflow/core/kernels:split_op_test_cpu                              PASSED in 0.8s
//tensorflow/core/kernels:split_v_op_test_cpu                            PASSED in 0.6s
//tensorflow/core/kernels:strided_slice_op_test                          PASSED in 0.4s
//tensorflow/core/kernels:string_format_op_test                          PASSED in 0.6s
//tensorflow/core/kernels:string_ngrams_op_test                          PASSED in 1.5s
//tensorflow/core/kernels:string_split_op_test                           PASSED in 0.5s
//tensorflow/core/kernels:substr_op_test                                 PASSED in 1.2s
//tensorflow/core/kernels:summary_audio_op_test                          PASSED in 0.6s
//tensorflow/core/kernels:summary_image_op_test                          PASSED in 7.7s
//tensorflow/core/kernels:summary_op_test                                PASSED in 10.3s
//tensorflow/core/kernels:summary_tensor_op_test                         PASSED in 0.6s
//tensorflow/core/kernels:tensor_cord_test                               PASSED in 0.3s
//tensorflow/core/kernels:tensor_flag_utils_test                         PASSED in 0.3s
//tensorflow/core/kernels:tensor_map_test                                PASSED in 0.1s
//tensorflow/core/kernels:training_ops_test                              PASSED in 0.4s
//tensorflow/core/kernels:transpose_util_test                            PASSED in 0.5s
//tensorflow/core/kernels:unary_ops_composition_test_cpu                 PASSED in 2.1s
//tensorflow/core/kernels:unique_op_test                                 PASSED in 0.5s
//tensorflow/core/kernels:variable_ops_test                              PASSED in 2.6s
//tensorflow/core/kernels:while_op_test                                  PASSED in 8.3s
//tensorflow/core/kernels:xent_op_test_cpu                               PASSED in 0.5s
//tensorflow/core/kernels/batching_util:basic_batch_scheduler_test       PASSED in 0.2s
//tensorflow/core/kernels/batching_util:batch_input_task_test            PASSED in 0.7s
//tensorflow/core/kernels/batching_util:batch_resource_base_test         PASSED in 0.2s
//tensorflow/core/kernels/batching_util:batch_scheduler_test             PASSED in 0.2s
//tensorflow/core/kernels/batching_util:bounded_executor_test            PASSED in 43.5s
//tensorflow/core/kernels/batching_util:input_split_metadata_test        PASSED in 0.2s
//tensorflow/core/kernels/batching_util:periodic_function_test           PASSED in 3.2s
//tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 2.2s
//tensorflow/core/kernels/batching_util:shared_batch_scheduler_test      PASSED in 5.3s
//tensorflow/core/kernels/batching_util:threadsafe_status_test           PASSED in 0.1s
//tensorflow/core/kernels/data:batch_dataset_op_test                     PASSED in 0.9s
//tensorflow/core/kernels/data:cache_dataset_ops_test                    PASSED in 0.8s
//tensorflow/core/kernels/data:concatenate_dataset_op_test               PASSED in 1.1s
//tensorflow/core/kernels/data:filter_dataset_op_test                    PASSED in 1.0s
//tensorflow/core/kernels/data:finalize_dataset_op_test                  PASSED in 0.8s
//tensorflow/core/kernels/data:fixed_length_record_dataset_op_test       PASSED in 0.9s
//tensorflow/core/kernels/data:flat_map_dataset_op_test                  PASSED in 2.0s
//tensorflow/core/kernels/data:get_options_op_test                       PASSED in 0.5s
//tensorflow/core/kernels/data:interleave_dataset_op_test                PASSED in 1.9s
//tensorflow/core/kernels/data:iterator_ops_test                         PASSED in 7.2s
//tensorflow/core/kernels/data:map_dataset_op_test                       PASSED in 0.8s
//tensorflow/core/kernels/data:map_defun_op_test                         PASSED in 0.6s
//tensorflow/core/kernels/data:optimize_dataset_op_test                  PASSED in 0.9s
//tensorflow/core/kernels/data:options_dataset_op_test                   PASSED in 0.9s
//tensorflow/core/kernels/data:padded_batch_dataset_op_test              PASSED in 27.2s
//tensorflow/core/kernels/data:parallel_batch_dataset_op_test            PASSED in 0.7s
//tensorflow/core/kernels/data:parallel_filter_dataset_op_test           PASSED in 1.1s
//tensorflow/core/kernels/data:parallel_interleave_dataset_op_test       PASSED in 1.3s
//tensorflow/core/kernels/data:parallel_map_dataset_op_test              PASSED in 2.0s
//tensorflow/core/kernels/data:prefetch_autotuner_test                   PASSED in 0.1s
//tensorflow/core/kernels/data:prefetch_dataset_op_test                  PASSED in 1.2s
//tensorflow/core/kernels/data:range_dataset_op_test                     PASSED in 0.7s
//tensorflow/core/kernels/data:reduce_dataset_op_test                    PASSED in 1.1s
//tensorflow/core/kernels/data:repeat_dataset_op_test                    PASSED in 1.2s
//tensorflow/core/kernels/data:rewrite_dataset_op_test                   PASSED in 1.2s
//tensorflow/core/kernels/data:shard_dataset_op_test                     PASSED in 0.6s
//tensorflow/core/kernels/data:shuffle_dataset_op_test                   PASSED in 1.0s
//tensorflow/core/kernels/data:skip_dataset_op_test                      PASSED in 0.8s
//tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test       PASSED in 1.0s
//tensorflow/core/kernels/data:take_dataset_op_test                      PASSED in 1.7s
//tensorflow/core/kernels/data:tensor_dataset_op_test                    PASSED in 0.7s
//tensorflow/core/kernels/data:tensor_slice_dataset_op_test              PASSED in 2.6s
//tensorflow/core/kernels/data:text_line_dataset_op_test                 PASSED in 1.4s
//tensorflow/core/kernels/data:tf_record_dataset_op_test                 PASSED in 4.6s
//tensorflow/core/kernels/data:window_dataset_op_test                    PASSED in 0.6s
//tensorflow/core/kernels/data:zip_dataset_op_test                       PASSED in 1.4s
//tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test  PASSED in 0.5s
//tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test  PASSED in 0.9s
//tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test   PASSED in 0.8s
//tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.5s
//tensorflow/core/kernels/data/experimental:list_dataset_op_test         PASSED in 0.6s
//tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 3.3s
//tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 1.0s
//tensorflow/core/kernels/data/experimental:random_dataset_op_test       PASSED in 0.7s
//tensorflow/core/kernels/data/experimental:sampling_dataset_op_test     PASSED in 0.8s
//tensorflow/core/kernels/data/experimental:save_dataset_op_test         PASSED in 1.8s
//tensorflow/core/kernels/data/experimental:unique_dataset_op_test       PASSED in 1.0s
//tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu    PASSED in 0.5s
//tensorflow/core/kernels/image:adjust_contrast_op_test                  PASSED in 16.5s
//tensorflow/core/kernels/image:colorspace_op_test                       PASSED in 0.6s
//tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu    PASSED in 0.6s
//tensorflow/core/kernels/image:crop_and_resize_op_test                  PASSED in 1.4s
//tensorflow/core/kernels/image:encode_jpeg_op_test                      PASSED in 0.5s
//tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu         PASSED in 0.6s
//tensorflow/core/kernels/image:mirror_pad_op_test                       PASSED in 0.7s
//tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test    PASSED in 0.9s
//tensorflow/core/kernels/image:non_max_suppression_op_test              PASSED in 0.9s
//tensorflow/core/kernels/image:resize_area_op_test                      PASSED in 1.7s
//tensorflow/core/kernels/image:resize_benchmark_test_cpu                PASSED in 0.5s
//tensorflow/core/kernels/image:resize_bicubic_op_test                   PASSED in 9.6s
//tensorflow/core/kernels/image:resize_ops_test_cpu                      PASSED in 2.4s
//tensorflow/core/kernels/image:sampling_kernels_test                    PASSED in 0.9s
//tensorflow/core/kernels/image:scale_and_translate_op_test              PASSED in 29.5s
//tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu     PASSED in 0.7s
//tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu     PASSED in 0.5s
//tensorflow/core/kernels/mkl:mkl_conv_ops_test                          PASSED in 0.1s
//tensorflow/core/kernels/mkl:mkl_dequantize_op_test                     PASSED in 0.2s
//tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test               PASSED in 0.4s
//tensorflow/core/kernels/mkl:mkl_fused_ops_test                         PASSED in 1.1s
//tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark                    PASSED in 0.1s
//tensorflow/core/kernels/mkl:mkl_qmatmul_op_test                        PASSED in 0.2s
//tensorflow/core/kernels/mkl:mkl_quantize_op_test                       PASSED in 0.1s
//tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test               PASSED in 0.3s
//tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test     PASSED in 0.4s
//tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test                PASSED in 0.1s
//tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test             PASSED in 0.1s
//tensorflow/core/kernels/mkl:mkl_relu_op_test                           PASSED in 0.1s
//tensorflow/core/kernels/mkl:mkl_requantize_ops_test                    PASSED in 0.2s
//tensorflow/core/kernels/mkl:mkl_swish_op_test                          PASSED in 0.9s
//tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark                    PASSED in 0.1s
//tensorflow/core/kernels/sparse:kernels_test                            PASSED in 0.6s
//tensorflow/core/kernels/uniform_quant_ops:math_utils_test              PASSED in 0.1s
//tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test            PASSED in 0.1s
//tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test   PASSED in 0.7s
//tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test     PASSED in 0.8s
//tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.6s
//tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.7s
//tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 16.0s
//tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 0.9s
//tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test   PASSED in 0.5s
//tensorflow/core/lib/db:sqlite_test                                     PASSED in 0.1s
//tensorflow/core/lib/gif:lib_gif_io_test                                PASSED in 1.1s
//tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest                    PASSED in 1.4s
//tensorflow/core/ops:cudnn_rnn_ops_test_cc                              PASSED in 0.6s
//tensorflow/core/ops:ops_array_grad_test                                PASSED in 1.4s
//tensorflow/core/ops:ops_math_grad_test                                 PASSED in 4.5s
//tensorflow/core/ops:ops_tests                                          PASSED in 0.6s
//tensorflow/core/ops/compat:backwards_compatibility_test                PASSED in 0.5s
//tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.1s
//tensorflow/core/platform:enable_tf2_utils_test                         PASSED in 0.1s
//tensorflow/core/platform:env_test                                      PASSED in 2.7s
//tensorflow/core/platform:fake_python_env_test                          PASSED in 0.1s
//tensorflow/core/platform:file_system_test                              PASSED in 0.5s
//tensorflow/core/platform:platform_strings_test                         PASSED in 0.1s
//tensorflow/core/platform:ram_file_system_test                          PASSED in 43.7s
//tensorflow/core/platform:resource_loader_test                          PASSED in 0.1s
//tensorflow/core/platform:vmodule_benchmark_test                        PASSED in 0.4s
//tensorflow/core/platform:vmodule_test                                  PASSED in 0.1s
//tensorflow/core/profiler/backends/cpu:host_tracer_test                 PASSED in 0.4s
//tensorflow/core/profiler/convert:dcn_analysis_test                     PASSED in 0.6s
//tensorflow/core/profiler/convert:dcn_utils_test                        PASSED in 0.6s
//tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test          PASSED in 0.2s
//tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.8s
//tensorflow/core/profiler/convert:op_stats_to_pod_stats_test            PASSED in 0.1s
//tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test           PASSED in 0.1s
//tensorflow/core/profiler/convert:op_stats_to_tf_stats_test             PASSED in 0.2s
//tensorflow/core/profiler/convert:repository_test                       PASSED in 0.1s
//tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test        PASSED in 0.2s
//tensorflow/core/profiler/convert:xplane_to_memory_profile_test         PASSED in 0.1s
//tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test          PASSED in 0.1s
//tensorflow/core/profiler/convert:xplane_to_op_stats_test               PASSED in 0.7s
//tensorflow/core/profiler/convert:xplane_to_step_events_test            PASSED in 0.1s
//tensorflow/core/profiler/convert:xplane_to_tf_functions_test           PASSED in 0.7s
//tensorflow/core/profiler/convert:xplane_to_tool_names_test             PASSED in 0.2s
//tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.1s
//tensorflow/core/profiler/internal:tfprof_show_test                     PASSED in 0.9s
//tensorflow/core/profiler/internal:tfprof_stats_test                    PASSED in 0.7s
//tensorflow/core/profiler/internal:tfprof_tensor_test                   PASSED in 0.6s
//tensorflow/core/profiler/internal:tfprof_timeline_test                 PASSED in 0.7s
//tensorflow/core/profiler/internal/advisor:tfprof_advisor_test          PASSED in 0.9s
//tensorflow/core/profiler/lib:profiler_disabled_test                    PASSED in 0.2s
//tensorflow/core/profiler/utils:derived_timeline_test                   PASSED in 0.1s
//tensorflow/core/profiler/utils:kernel_stats_utils_test                 PASSED in 0.4s
//tensorflow/core/profiler/utils:op_metrics_db_utils_test                PASSED in 0.1s
//tensorflow/core/profiler/utils:step_intersection_test                  PASSED in 0.1s
//tensorflow/core/runtime_fallback/util:type_util_test                   PASSED in 0.1s
//tensorflow/core/summary:schema_test                                    PASSED in 0.1s
//tensorflow/core/summary:summary_db_writer_test                         PASSED in 0.9s
//tensorflow/core/summary:summary_file_writer_test                       PASSED in 0.1s
//tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test          PASSED in 5.7s
//tensorflow/core/tfrt/common:pjrt_state_test                            PASSED in 8.1s
//tensorflow/core/tfrt/common:pjrt_util_test                             PASSED in 6.5s
//tensorflow/core/tfrt/fallback:cost_recorder_test                       PASSED in 1.0s
//tensorflow/core/tfrt/fallback:fallback_state_test                      PASSED in 0.4s
//tensorflow/core/tfrt/graph_executor:config_test                        PASSED in 0.2s
//tensorflow/core/tfrt/mlrt/attribute:attribute_test                     PASSED in 1.0s
//tensorflow/core/tfrt/mlrt/bytecode:bytecode_test                       PASSED in 0.2s
//tensorflow/core/tfrt/mlrt/bytecode:executable_test                     PASSED in 0.2s
//tensorflow/core/tfrt/mlrt/bytecode:function_test                       PASSED in 0.1s
//tensorflow/core/tfrt/mlrt/bytecode:kernel_test                         PASSED in 0.2s
//tensorflow/core/tfrt/mlrt/bytecode:span_test                           PASSED in 0.2s
//tensorflow/core/tfrt/mlrt/interpreter:context_test                     PASSED in 0.1s
//tensorflow/core/tfrt/mlrt/interpreter:future_test                      PASSED in 0.1s
//tensorflow/core/tfrt/mlrt/interpreter:interpreter_test                 PASSED in 0.1s
//tensorflow/core/tfrt/mlrt/interpreter:register_span_test               PASSED in 0.1s
//tensorflow/core/tfrt/mlrt/interpreter:value_test                       PASSED in 0.1s
//tensorflow/core/tfrt/run_handler_thread_pool:run_handler_concurrent_work_queue_test PASSED in 0.4s
//tensorflow/core/tfrt/run_handler_thread_pool:run_handler_test          PASSED in 1.4s
//tensorflow/core/tfrt/run_handler_thread_pool:run_handler_util_test     PASSED in 0.2s
//tensorflow/core/tfrt/runtime:channel_test                              PASSED in 0.2s
//tensorflow/core/tfrt/runtime:tf_threadpool_concurrent_work_queue_test  PASSED in 0.5s
//tensorflow/core/tfrt/runtime:work_queue_interface_test                 PASSED in 0.3s
//tensorflow/core/tfrt/utils:graph_partition_test                        PASSED in 3.2s
//tensorflow/core/transforms:eval_utils_test                             PASSED in 1.9s
//tensorflow/core/transforms:graph_transform_wrapper_test                PASSED in 0.2s
//tensorflow/core/util:bcast_test                                        PASSED in 16.4s
//tensorflow/core/util:command_line_flags_test                           PASSED in 0.7s
//tensorflow/core/util:debug_data_dumper_test                            PASSED in 1.2s
//tensorflow/core/util:debug_events_writer_test                          PASSED in 0.3s
//tensorflow/core/util:dump_graph_test                                   PASSED in 1.0s
//tensorflow/core/util:equal_graph_def_test                              PASSED in 7.8s
//tensorflow/core/util:events_writer_test                                PASSED in 3.2s
//tensorflow/core/util:example_proto_fast_parsing_test                   PASSED in 1.6s
//tensorflow/core/util:example_proto_helper_test                         PASSED in 0.8s
//tensorflow/core/util:exec_on_stall_test                                PASSED in 2.1s
//tensorflow/core/util:fake_clock_env_test                               PASSED in 2.2s
//tensorflow/core/util:incremental_barrier_test                          PASSED in 0.2s
//tensorflow/core/util:matmul_bcast_test                                 PASSED in 0.8s
//tensorflow/core/util:memmapped_file_system_test                        PASSED in 0.8s
//tensorflow/core/util:mkl_heuristics_test                               PASSED in 0.1s
//tensorflow/core/util:overflow_test                                     PASSED in 0.1s
//tensorflow/core/util:presized_cuckoo_map_test                          PASSED in 2.0s
//tensorflow/core/util:ragged_to_dense_util_test                         PASSED in 0.5s
//tensorflow/core/util:reffed_status_callback_test                       PASSED in 0.8s
//tensorflow/core/util:reporter_test                                     PASSED in 1.2s
//tensorflow/core/util:saved_tensor_slice_util_test                      PASSED in 0.9s
//tensorflow/core/util:semver_test                                       PASSED in 0.9s
//tensorflow/core/util:stat_summarizer_test                              PASSED in 1.1s
//tensorflow/core/util:strided_slice_op_test                             PASSED in 1.8s
//tensorflow/core/util:tensor_format_test                                PASSED in 10.4s
//tensorflow/core/util:tensor_slice_reader_test                          PASSED in 1.0s
//tensorflow/core/util:tensor_slice_set_test                             PASSED in 1.1s
//tensorflow/core/util:tensor_slice_util_test                            PASSED in 1.1s
//tensorflow/core/util:tensor_slice_writer_test                          PASSED in 1.5s
//tensorflow/core/util:work_sharder_test                                 PASSED in 1.1s
//tensorflow/core/util/ctc:ctc_beam_search_test                          PASSED in 0.1s
//tensorflow/core/util/proto:descriptor_pool_registry_test               PASSED in 0.6s
//tensorflow/core/util/proto:proto_utils_test                            PASSED in 0.5s
//tensorflow/core/util/quantization:uniform_quant_ops_params_test        PASSED in 1.0s
//tensorflow/core/util/sparse:sparse_tensor_test                         PASSED in 0.1s
//tensorflow/core/util/tensor_bundle:tensor_bundle_test                  PASSED in 32.0s
//tensorflow/dtensor/mlir:dtensor_location_test                          PASSED in 0.2s
//tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test          PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test    PASSED in 0.8s
//tensorflow/dtensor/mlir/tests:constant_folding.mlir.test               PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:decompose_controlflow.mlir.test          PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 1.1s
//tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test             PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test            PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 1.7s
//tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test     PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test      PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:dtensor_collective_type_lowering.mlir.test PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test    PASSED in 0.8s
//tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 2.2s
//tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 2.6s
//tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test   PASSED in 1.6s
//tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test       PASSED in 0.8s
//tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 1.5s
//tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test   PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:function_renaming.mlir.test              PASSED in 1.8s
//tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test           PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test          PASSED in 1.4s
//tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test                PASSED in 1.2s
//tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test                 PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test               PASSED in 1.2s
//tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test         PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test           PASSED in 1.0s
//tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test       PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.8s
//tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test             PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test        PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test           PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test               PASSED in 0.8s
//tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test             PASSED in 1.7s
//tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test                    PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test                      PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test                    PASSED in 2.0s
//tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test                 PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:spmd_fft.mlir.test                       PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test                    PASSED in 1.5s
//tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test                  PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test                    PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:spmd_random.mlir.test                    PASSED in 0.6s
//tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test              PASSED in 1.8s
//tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test               PASSED in 1.9s
//tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test                     PASSED in 0.8s
//tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test              PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test                   PASSED in 0.8s
//tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test                PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test                 PASSED in 0.7s
//tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.9s
//tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test                PASSED in 1.2s
//tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test   PASSED in 1.8s
//tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test            PASSED in 0.9s
//tensorflow/dtensor/python/tests:array_ops_test_cpu                     PASSED in 40.1s
//tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 23.3s
//tensorflow/dtensor/python/tests:collective_test_cpu                    PASSED in 28.2s
//tensorflow/dtensor/python/tests:config_test_cpu                        PASSED in 10.8s
//tensorflow/dtensor/python/tests:device_test_cpu                        PASSED in 46.0s
//tensorflow/dtensor/python/tests:layout_test_cpu                        PASSED in 20.1s
//tensorflow/dtensor/python/tests:multi_client_test_cpu                  PASSED in 17.8s
//tensorflow/dtensor/python/tests:numpy_util_test_cpu                    PASSED in 12.2s
//tensorflow/dtensor/python/tests:variable_test_cpu                      PASSED in 15.7s
//tensorflow/dtensor/tests:dtensor_operation_test                        PASSED in 33.8s
//tensorflow/dtensor/tests:executable_manager_test                       PASSED in 34.8s
//tensorflow/dtensor/tests:layout_to_xla_sharding_test                   PASSED in 0.2s
//tensorflow/dtensor/tests:slice_util_test                               PASSED in 0.1s
//tensorflow/dtensor/tests:spmd_expander_test                            PASSED in 7.7s
//tensorflow/dtensor/tests:tensor_layout_test                            PASSED in 0.2s
//tensorflow/examples/adding_an_op:fact_test                             PASSED in 21.9s
//tensorflow/examples/adding_an_op:zero_out_1_test                       PASSED in 22.6s
//tensorflow/examples/adding_an_op:zero_out_2_test                       PASSED in 24.0s
//tensorflow/examples/adding_an_op:zero_out_3_test                       PASSED in 21.9s
//tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test        PASSED in 43.2s
//tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu    PASSED in 32.6s
//tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test        PASSED in 46.2s
//tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test        PASSED in 46.5s
//tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 29.3s
//tensorflow/examples/custom_ops_doc/sleep:sleep_test                    PASSED in 24.3s
//tensorflow/examples/speech_commands:accuracy_utils_test                PASSED in 3.3s
//tensorflow/examples/speech_commands:models_test                        PASSED in 64.6s
//tensorflow/examples/speech_commands:recognize_commands_test            PASSED in 1.8s
//tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test         PASSED in 2.2s
//tensorflow/js:ts_op_gen_test                                           PASSED in 0.4s
//tensorflow/python/autograph/converters:asserts_test                    PASSED in 16.6s
//tensorflow/python/autograph/converters:break_statements_test           PASSED in 11.0s
//tensorflow/python/autograph/converters:call_trees_test                 PASSED in 9.8s
//tensorflow/python/autograph/converters:conditional_expressions_test    PASSED in 12.7s
//tensorflow/python/autograph/converters:continue_statements_test        PASSED in 49.1s
//tensorflow/python/autograph/converters:control_flow_test               PASSED in 17.3s
//tensorflow/python/autograph/converters:directives_test                 PASSED in 10.1s
//tensorflow/python/autograph/converters:functions_test                  PASSED in 17.0s
//tensorflow/python/autograph/converters:lists_test                      PASSED in 10.6s
//tensorflow/python/autograph/converters:logical_expressions_test        PASSED in 12.2s
//tensorflow/python/autograph/converters:return_statements_test          PASSED in 13.9s
//tensorflow/python/autograph/converters:slices_test                     PASSED in 39.0s
//tensorflow/python/autograph/converters:variables_test                  PASSED in 10.4s
//tensorflow/python/autograph/core:converter_test                        PASSED in 28.9s
//tensorflow/python/autograph/core:function_wrappers_test                PASSED in 12.7s
//tensorflow/python/autograph/impl:api_test                              PASSED in 16.4s
//tensorflow/python/autograph/impl:conversion_test                       PASSED in 11.9s
//tensorflow/python/autograph/lang:special_functions_test                PASSED in 15.8s
//tensorflow/python/autograph/operators:conditional_expressions_test     PASSED in 19.8s
//tensorflow/python/autograph/operators:control_flow_test                PASSED in 29.6s
//tensorflow/python/autograph/operators:data_structures_test             PASSED in 40.8s
//tensorflow/python/autograph/operators:exceptions_test                  PASSED in 10.1s
//tensorflow/python/autograph/operators:logical_test                     PASSED in 23.4s
//tensorflow/python/autograph/operators:py_builtins_test                 PASSED in 18.1s
//tensorflow/python/autograph/operators:slices_test                      PASSED in 15.9s
//tensorflow/python/autograph/operators:variables_test                   PASSED in 9.8s
//tensorflow/python/autograph/pyct:anno_test                             PASSED in 10.0s
//tensorflow/python/autograph/pyct:ast_util_test                         PASSED in 9.8s
//tensorflow/python/autograph/pyct:cache_test                            PASSED in 13.2s
//tensorflow/python/autograph/pyct:cfg_test                              PASSED in 10.2s
//tensorflow/python/autograph/pyct:error_utils_test                      PASSED in 18.8s
//tensorflow/python/autograph/pyct:inspect_utils_test                    PASSED in 14.4s
//tensorflow/python/autograph/pyct:loader_test                           PASSED in 9.9s
//tensorflow/python/autograph/pyct:naming_test                           PASSED in 12.8s
//tensorflow/python/autograph/pyct:origin_info_test                      PASSED in 18.0s
//tensorflow/python/autograph/pyct:parser_test                           PASSED in 10.5s
//tensorflow/python/autograph/pyct:pretty_printer_test                   PASSED in 19.5s
//tensorflow/python/autograph/pyct:qual_names_test                       PASSED in 39.7s
//tensorflow/python/autograph/pyct:templates_test                        PASSED in 9.7s
//tensorflow/python/autograph/pyct:transformer_test                      PASSED in 17.8s
//tensorflow/python/autograph/pyct:transpiler_test                       PASSED in 9.6s
//tensorflow/python/autograph/pyct/static_analysis:activity_test         PASSED in 11.6s
//tensorflow/python/autograph/pyct/static_analysis:liveness_test         PASSED in 10.5s
//tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 10.6s
//tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test  PASSED in 24.3s
//tensorflow/python/autograph/pyct/static_analysis:type_inference_test   PASSED in 12.9s
//tensorflow/python/autograph/tests:assertion_test                       PASSED in 29.8s
//tensorflow/python/autograph/tests:basic_ifexp_test                     PASSED in 23.0s
//tensorflow/python/autograph/tests:call_to_builtin_function_test        PASSED in 29.7s
//tensorflow/python/autograph/tests:call_to_lambda_function_test         PASSED in 24.1s
//tensorflow/python/autograph/tests:call_to_named_tuple_test             PASSED in 23.6s
//tensorflow/python/autograph/tests:call_to_numpy_function_test          PASSED in 30.5s
//tensorflow/python/autograph/tests:call_to_print_function_test          PASSED in 27.0s
//tensorflow/python/autograph/tests:call_to_tf_api_test                  PASSED in 22.1s
//tensorflow/python/autograph/tests:call_to_user_function_test           PASSED in 22.6s
//tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 53.4s
//tensorflow/python/autograph/tests:cond_basic_test                      PASSED in 35.4s
//tensorflow/python/autograph/tests:datasets_test                        PASSED in 25.6s
//tensorflow/python/autograph/tests:early_return_test                    PASSED in 28.4s
//tensorflow/python/autograph/tests:ext_slice_test                       PASSED in 48.0s
//tensorflow/python/autograph/tests:generator_test                       PASSED in 22.4s
//tensorflow/python/autograph/tests:logical_expression_test              PASSED in 27.3s
//tensorflow/python/autograph/tests:loop_basic_test                      PASSED in 85.4s
//tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 23.2s
//tensorflow/python/autograph/tests:loop_created_variables_test          PASSED in 34.0s
//tensorflow/python/autograph/tests:loop_scoping_test                    PASSED in 65.3s
//tensorflow/python/autograph/tests:loop_with_function_call_test         PASSED in 107.9s
//tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 29.5s
//tensorflow/python/autograph/tests:loop_with_variable_type_test         PASSED in 52.3s
//tensorflow/python/autograph/tests:nested_control_flow_test             PASSED in 74.9s
//tensorflow/python/autograph/tests:type_annotations_test                PASSED in 21.9s
//tensorflow/python/autograph/utils:context_managers_test                PASSED in 12.0s
//tensorflow/python/autograph/utils:misc_test                            PASSED in 11.5s
//tensorflow/python/autograph/utils:tensor_list_test                     PASSED in 11.0s
//tensorflow/python/autograph/utils:tensors_test                         PASSED in 12.8s
//tensorflow/python/checkpoint:benchmarks_test                           PASSED in 19.5s
//tensorflow/python/checkpoint:checkpoint_management_test_cpu            PASSED in 18.9s
//tensorflow/python/checkpoint:checkpoint_metrics_test                   PASSED in 16.9s
//tensorflow/python/checkpoint:checkpoint_test                           PASSED in 43.4s
//tensorflow/python/checkpoint:checkpoint_view_test                      PASSED in 12.8s
//tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test        PASSED in 13.7s
//tensorflow/python/checkpoint:functional_saver_test_cpu                 PASSED in 14.6s
//tensorflow/python/checkpoint:restore_test                              PASSED in 12.5s
//tensorflow/python/checkpoint:save_util_v1_test                         PASSED in 14.3s
//tensorflow/python/checkpoint:saveable_compat_test                      PASSED in 11.5s
//tensorflow/python/checkpoint:tensor_callable_test                      PASSED in 18.5s
//tensorflow/python/checkpoint:trackable_view_test                       PASSED in 12.8s
//tensorflow/python/client:device_lib_test_cpu                           PASSED in 16.4s
//tensorflow/python/client:events_writer_test                            PASSED in 11.5s
//tensorflow/python/client:session_benchmark_cpu                         PASSED in 15.7s
//tensorflow/python/client:session_list_devices_test                     PASSED in 13.1s
//tensorflow/python/client:session_partial_run_test                      PASSED in 16.7s
//tensorflow/python/client:timeline_test_cpu                             PASSED in 34.3s
//tensorflow/python/client:virtual_gpu_test_cpu                          PASSED in 14.2s
//tensorflow/python/compat:compat_test                                   PASSED in 10.3s
//tensorflow/python/compat:disable_v2_behavior_test                      PASSED in 26.3s
//tensorflow/python/compiler/mlir:mlir_test                              PASSED in 10.5s
//tensorflow/python/compiler/tensorrt:trt_convert_test_cpu               PASSED in 11.1s
//tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu         PASSED in 11.6s
//tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu       PASSED in 11.1s
//tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 14.2s
//tensorflow/python/compiler/tensorrt/test:bool_test_cpu                 PASSED in 11.3s
//tensorflow/python/compiler/tensorrt/test:cast_test_cpu                 PASSED in 11.6s
//tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu        PASSED in 10.5s
//tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu      PASSED in 11.4s
//tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 11.6s
//tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 11.0s
//tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu      PASSED in 11.7s
//tensorflow/python/compiler/tensorrt/test:int32_test_cpu                PASSED in 11.1s
//tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu            PASSED in 13.4s
//tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 27.7s
//tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu   PASSED in 13.4s
//tensorflow/python/compiler/tensorrt/test:quantization_test_cpu         PASSED in 11.0s
//tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu             PASSED in 11.5s
//tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu    PASSED in 11.2s
//tensorflow/python/compiler/tensorrt/test:topk_test_cpu                 PASSED in 10.9s
//tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu  PASSED in 12.2s
//tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu             PASSED in 17.8s
//tensorflow/python/compiler/tensorrt/test:unary_test_cpu                PASSED in 11.6s
//tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu       PASSED in 14.3s
//tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu            PASSED in 17.7s
//tensorflow/python/compiler/xla:jit_compile_test_cpu                    PASSED in 11.4s
//tensorflow/python/compiler/xla:jit_test_cpu                            PASSED in 17.6s
//tensorflow/python/compiler/xla:xla_test_cpu                            PASSED in 43.2s
//tensorflow/python/compiler/xla/experimental:xla_sharding_test          PASSED in 13.0s
//tensorflow/python/data/benchmarks:batch_benchmark                      PASSED in 10.3s
//tensorflow/python/data/benchmarks:filter_benchmark                     PASSED in 14.7s
//tensorflow/python/data/benchmarks:from_tensor_slices_benchmark         PASSED in 11.3s
//tensorflow/python/data/benchmarks:interleave_benchmark                 PASSED in 14.0s
//tensorflow/python/data/benchmarks:list_files_benchmark                 PASSED in 9.6s
//tensorflow/python/data/benchmarks:map_benchmark                        PASSED in 10.9s
//tensorflow/python/data/benchmarks:meta_benchmark                       PASSED in 10.0s
//tensorflow/python/data/benchmarks:prefetch_benchmark                   PASSED in 23.5s
//tensorflow/python/data/benchmarks:range_benchmark                      PASSED in 14.9s
//tensorflow/python/data/experimental/benchmarks:autotune_benchmark      PASSED in 11.4s
//tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark   PASSED in 12.7s
//tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 12.3s
//tensorflow/python/data/experimental/benchmarks:map_defun_benchmark     PASSED in 10.6s
//tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 11.0s
//tensorflow/python/data/experimental/benchmarks:optimize_benchmark      PASSED in 15.7s
//tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 9.9s
//tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 10.8s
//tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 10.8s
//tensorflow/python/data/experimental/benchmarks:unbatch_benchmark       PASSED in 13.6s
//tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 43.7s
//tensorflow/python/data/experimental/kernel_tests:assert_next_test      PASSED in 11.8s
//tensorflow/python/data/experimental/kernel_tests:assert_prev_test      PASSED in 14.4s
//tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 69.1s
//tensorflow/python/data/experimental/kernel_tests:compression_ops_test  PASSED in 24.7s
//tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 27.4s
//tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 21.9s
//tensorflow/python/data/experimental/kernel_tests:from_list_test        PASSED in 37.7s
//tensorflow/python/data/experimental/kernel_tests:io_test               PASSED in 68.7s
//tensorflow/python/data/experimental/kernel_tests:lookup_ops_test       PASSED in 13.5s
//tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 27.0s
//tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 11.9s
//tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 60.7s
//tensorflow/python/data/experimental/kernel_tests:map_defun_op_test     PASSED in 11.8s
//tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 18.7s
//tensorflow/python/data/experimental/kernel_tests:model_dataset_test    PASSED in 11.8s
//tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 13.7s
//tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 13.3s
//tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 13.3s
//tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 12.6s
//tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 24.7s
//tensorflow/python/data/experimental/kernel_tests:sleep_test            PASSED in 21.6s
//tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 13.1s
//tensorflow/python/data/experimental/kernel_tests:variant_test          PASSED in 11.0s
//tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu  PASSED in 10.5s
//tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 37.9s
//tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 59.9s
//tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 13.8s
//tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 38.8s
//tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 13.5s
//tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 45.1s
//tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 23.2s
//tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 16.2s
//tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 14.9s
//tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 30.8s
//tensorflow/python/data/experimental/service:server_lib_test            PASSED in 12.2s
//tensorflow/python/data/kernel_tests:as_numpy_iterator_test             PASSED in 13.5s
//tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test     PASSED in 25.0s
//tensorflow/python/data/kernel_tests:cache_test                         PASSED in 85.8s
//tensorflow/python/data/kernel_tests:cardinality_test                   PASSED in 21.9s
//tensorflow/python/data/kernel_tests:checkpoint_test                    PASSED in 20.6s
//tensorflow/python/data/kernel_tests:concatenate_test                   PASSED in 31.6s
//tensorflow/python/data/kernel_tests:counter_test                       PASSED in 36.2s
//tensorflow/python/data/kernel_tests:dataset_spec_test                  PASSED in 12.1s
//tensorflow/python/data/kernel_tests:dataset_test                       PASSED in 32.1s
//tensorflow/python/data/kernel_tests:enumerate_test                     PASSED in 30.2s
//tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test     PASSED in 10.6s
//tensorflow/python/data/kernel_tests:from_tensor_slices_test            PASSED in 36.4s
//tensorflow/python/data/kernel_tests:from_tensors_test                  PASSED in 21.0s
//tensorflow/python/data/kernel_tests:get_single_element_test            PASSED in 15.1s
//tensorflow/python/data/kernel_tests:ignore_errors_test                 PASSED in 20.6s
//tensorflow/python/data/kernel_tests:io_test                            PASSED in 124.7s
//tensorflow/python/data/kernel_tests:iterator_test_cpu                  PASSED in 39.4s
//tensorflow/python/data/kernel_tests:len_test                           PASSED in 12.9s
//tensorflow/python/data/kernel_tests:list_files_test                    PASSED in 13.0s
//tensorflow/python/data/kernel_tests:optional_test_cpu                  PASSED in 22.8s
//tensorflow/python/data/kernel_tests:options_test                       PASSED in 11.4s
//tensorflow/python/data/kernel_tests:placement_test_cpu                 PASSED in 36.4s
//tensorflow/python/data/kernel_tests:prefetch_test                      PASSED in 49.0s
//tensorflow/python/data/kernel_tests:random_test                        PASSED in 30.5s
//tensorflow/python/data/kernel_tests:range_test                         PASSED in 42.4s
//tensorflow/python/data/kernel_tests:rebatch_test                       PASSED in 10.2s
//tensorflow/python/data/kernel_tests:reduce_test_cpu                    PASSED in 25.8s
//tensorflow/python/data/kernel_tests:scan_test_cpu                      PASSED in 56.2s
//tensorflow/python/data/kernel_tests:sparse_batch_test                  PASSED in 27.8s
//tensorflow/python/data/kernel_tests:unbatch_test                       PASSED in 65.6s
//tensorflow/python/data/util:convert_test                               PASSED in 11.7s
//tensorflow/python/data/util:nest_test                                  PASSED in 11.6s
//tensorflow/python/data/util:options_test                               PASSED in 9.8s
//tensorflow/python/data/util:random_seed_test                           PASSED in 97.7s
//tensorflow/python/data/util:sparse_test                                PASSED in 28.5s
//tensorflow/python/data/util:structure_test                             PASSED in 13.0s
//tensorflow/python/data/util:traverse_test                              PASSED in 23.5s
//tensorflow/python/debug/cli:analyzer_cli_test_cpu                      PASSED in 14.8s
//tensorflow/python/debug/cli:cli_config_test                            PASSED in 13.3s
//tensorflow/python/debug/cli:cli_shared_test                            PASSED in 10.9s
//tensorflow/python/debug/cli:command_parser_test                        PASSED in 9.2s
//tensorflow/python/debug/cli:debugger_cli_common_test                   PASSED in 9.5s
//tensorflow/python/debug/cli:evaluator_test                             PASSED in 11.1s
//tensorflow/python/debug/cli:profile_analyzer_cli_test                  PASSED in 9.9s
//tensorflow/python/debug/cli:readline_ui_test                           PASSED in 9.8s
//tensorflow/python/debug/cli:tensor_format_test                         PASSED in 10.0s
//tensorflow/python/debug/lib:check_numerics_callback_test_cpu           PASSED in 15.4s
//tensorflow/python/debug/lib:common_test                                PASSED in 11.3s
//tensorflow/python/debug/lib:debug_data_test                            PASSED in 30.2s
//tensorflow/python/debug/lib:debug_events_monitors_test                 PASSED in 13.8s
//tensorflow/python/debug/lib:debug_events_writer_test                   PASSED in 19.3s
//tensorflow/python/debug/lib:debug_gradients_test_cpu                   PASSED in 10.4s
//tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu        PASSED in 11.7s
//tensorflow/python/debug/lib:debug_graphs_test                          PASSED in 11.4s
//tensorflow/python/debug/lib:debug_grappler_test_cpu                    PASSED in 33.6s
//tensorflow/python/debug/lib:debug_utils_test                           PASSED in 10.5s
//tensorflow/python/debug/lib:debug_v2_ops_test_cpu                      PASSED in 20.2s
//tensorflow/python/debug/lib:profiling_test                             PASSED in 10.7s
//tensorflow/python/debug/lib:session_debug_file_test_cpu                PASSED in 16.4s
//tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu           PASSED in 13.1s
//tensorflow/python/debug/lib:source_utils_test                          PASSED in 17.3s
//tensorflow/python/debug/wrappers:disk_usage_test                       PASSED in 10.1s
//tensorflow/python/debug/wrappers:dumping_wrapper_test                  PASSED in 14.6s
//tensorflow/python/debug/wrappers:framework_test                        PASSED in 13.3s
//tensorflow/python/debug/wrappers:local_cli_wrapper_test                PASSED in 11.7s
//tensorflow/python/distribute:checkpoint_utils_test_2gpu                PASSED in 14.8s
//tensorflow/python/distribute:checkpoint_utils_test_cpu                 PASSED in 30.7s
//tensorflow/python/distribute:checkpointing_test_2gpu                   PASSED in 13.3s
//tensorflow/python/distribute:checkpointing_test_cpu                    PASSED in 16.2s
//tensorflow/python/distribute:collective_util_test                      PASSED in 10.7s
//tensorflow/python/distribute:combinations_test_2gpu                    PASSED in 24.9s
//tensorflow/python/distribute:combinations_test_cpu                     PASSED in 23.6s
//tensorflow/python/distribute:cross_device_utils_test_cpu               PASSED in 12.4s
//tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu   PASSED in 16.9s
//tensorflow/python/distribute:custom_training_loop_gradient_test_cpu    PASSED in 16.0s
//tensorflow/python/distribute:device_util_test_cpu                      PASSED in 28.5s
//tensorflow/python/distribute:distribute_coordinator_test               PASSED in 27.2s
//tensorflow/python/distribute:distribute_lib_test                       PASSED in 14.8s
//tensorflow/python/distribute:distribute_utils_test_2gpu                PASSED in 21.9s
//tensorflow/python/distribute:distribute_utils_test_cpu                 PASSED in 14.5s
//tensorflow/python/distribute:input_ops_test_cpu                        PASSED in 16.7s
//tensorflow/python/distribute:metrics_v1_test_2gpu                      PASSED in 34.0s
//tensorflow/python/distribute:metrics_v1_test_cpu                       PASSED in 63.9s
//tensorflow/python/distribute:mirrored_values_test_2gpu                 PASSED in 12.7s
//tensorflow/python/distribute:mirrored_values_test_cpu                  PASSED in 12.5s
//tensorflow/python/distribute:mirrored_variable_test_2gpu               PASSED in 48.5s
//tensorflow/python/distribute:mirrored_variable_test_cpu                PASSED in 36.5s
//tensorflow/python/distribute:multi_process_runner_no_init_test         PASSED in 21.3s
//tensorflow/python/distribute:multi_worker_continuous_run_test_cpu      PASSED in 27.7s
//tensorflow/python/distribute:multi_worker_util_test                    PASSED in 10.2s
//tensorflow/python/distribute:numpy_dataset_test                        PASSED in 10.1s
//tensorflow/python/distribute:one_device_strategy_test_cpu              PASSED in 21.7s
//tensorflow/python/distribute:packed_distributed_variable_test          PASSED in 11.0s
//tensorflow/python/distribute:parameter_server_strategy_test_2gpu       PASSED in 36.0s
//tensorflow/python/distribute:parameter_server_strategy_test_cpu        PASSED in 63.9s
//tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu    PASSED in 26.8s
//tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu     PASSED in 45.4s
//tensorflow/python/distribute:per_replica_test_2gpu                     PASSED in 15.8s
//tensorflow/python/distribute:per_replica_test_cpu                      PASSED in 13.8s
//tensorflow/python/distribute:ps_values_test_2gpu                       PASSED in 12.7s
//tensorflow/python/distribute:ps_values_test_cpu                        PASSED in 18.4s
//tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu   PASSED in 11.7s
//tensorflow/python/distribute:sharded_variable_test                     PASSED in 35.7s
//tensorflow/python/distribute:shared_variable_creator_test              PASSED in 21.8s
//tensorflow/python/distribute:strategy_combinations_test_cpu            PASSED in 55.2s
//tensorflow/python/distribute:template_mirrored_strategy_test_cpu       PASSED in 15.0s
//tensorflow/python/distribute:test_util_test_2gpu                       PASSED in 21.1s
//tensorflow/python/distribute:test_util_test_cpu                        PASSED in 51.8s
//tensorflow/python/distribute:tf_function_test_2gpu                     PASSED in 13.6s
//tensorflow/python/distribute:tf_function_test_cpu                      PASSED in 30.9s
//tensorflow/python/distribute:values_v2_test_cpu                        PASSED in 15.6s
//tensorflow/python/distribute:warm_starting_util_test_2gpu              PASSED in 14.3s
//tensorflow/python/distribute:warm_starting_util_test_cpu               PASSED in 15.3s
//tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 11.6s
//tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 14.3s
//tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 12.2s
//tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 13.0s
//tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 11.2s
//tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 11.8s
//tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 13.4s
//tensorflow/python/distribute/coordinator:watchdog_test                 PASSED in 65.0s
//tensorflow/python/distribute/experimental:dtensor_util_test_cpu        PASSED in 29.7s
//tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu   PASSED in 33.1s
//tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 21.8s
//tensorflow/python/distribute/integration_test:saved_model_test_cpu     PASSED in 57.3s
//tensorflow/python/distribute/parallel_device:parallel_device_test_cpu  PASSED in 14.2s
//tensorflow/python/distribute/v1:all_reduce_test                        PASSED in 51.3s
//tensorflow/python/distribute/v1:cross_device_ops_test_2gpu             PASSED in 65.9s
//tensorflow/python/distribute/v1:cross_device_ops_test_cpu              PASSED in 76.4s
//tensorflow/python/dlpack:dlpack_test_cpu                               PASSED in 20.4s
//tensorflow/python/eager:backprop_test_cpu                              PASSED in 147.1s
//tensorflow/python/eager:benchmarks_test_cpu                            PASSED in 11.9s
//tensorflow/python/eager:cancellation_test_cpu                          PASSED in 10.8s
//tensorflow/python/eager:context_test_cpu                               PASSED in 35.4s
//tensorflow/python/eager:core_test_cpu                                  PASSED in 20.2s
//tensorflow/python/eager:gradient_input_output_exclusions_test          PASSED in 47.9s
//tensorflow/python/eager:graph_only_ops_test_cpu                        PASSED in 10.1s
//tensorflow/python/eager:lift_to_graph_test                             PASSED in 15.8s
//tensorflow/python/eager:monitoring_test_cpu                            PASSED in 13.8s
//tensorflow/python/eager:ops_test_cpu                                   PASSED in 18.1s
//tensorflow/python/eager:profiler_client_test                           PASSED in 9.7s
//tensorflow/python/eager:profiler_test_cpu                              PASSED in 11.6s
//tensorflow/python/eager:pywrap_tfe_test                                PASSED in 42.4s
//tensorflow/python/eager:record_test                                    PASSED in 11.0s
//tensorflow/python/eager:remote_benchmarks_test_cpu                     PASSED in 9.6s
//tensorflow/python/eager:run_eager_op_as_function_test_cpu              PASSED in 14.6s
//tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu          PASSED in 9.5s
//tensorflow/python/eager:small_constants_optimizer_test_cpu             PASSED in 225.7s
//tensorflow/python/eager:tensor_test_cpu                                PASSED in 20.6s
//tensorflow/python/eager:wrap_function_device_test_cpu                  PASSED in 10.7s
//tensorflow/python/eager:wrap_function_test                             PASSED in 11.8s
//tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu              PASSED in 20.3s
//tensorflow/python/eager/memory_tests:remote_memory_test_cpu            PASSED in 10.0s
//tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu  PASSED in 12.4s
//tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu  PASSED in 11.9s
//tensorflow/python/eager/polymorphic_function:collection_test_cpu       PASSED in 10.6s
//tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu      PASSED in 10.2s
//tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 12.6s
//tensorflow/python/eager/polymorphic_function:concrete_function_test_cpu PASSED in 14.7s
//tensorflow/python/eager/polymorphic_function:function_spec_test        PASSED in 41.0s
//tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu PASSED in 28.1s
//tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu_mlir_bridge_test PASSED in 27.9s
//tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 10.3s
//tensorflow/python/eager/polymorphic_function:tracing_compilation_test  PASSED in 26.5s
//tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 38.5s
//tensorflow/python/feature_column:serialization_test                    PASSED in 21.7s
//tensorflow/python/framework:auto_control_deps_test                     PASSED in 35.9s
//tensorflow/python/framework:c_api_util_test                            PASSED in 12.9s
//tensorflow/python/framework:common_shapes_test                         PASSED in 11.9s
//tensorflow/python/framework:composite_tensor_test                      PASSED in 12.4s
//tensorflow/python/framework:config_test_2gpu                           PASSED in 17.7s
//tensorflow/python/framework:config_test_cpu                            PASSED in 15.4s
//tensorflow/python/framework:constant_op_test                           PASSED in 15.6s
//tensorflow/python/framework:device_spec_test                           PASSED in 14.5s
//tensorflow/python/framework:device_test                                PASSED in 10.9s
//tensorflow/python/framework:dtypes_test                                PASSED in 56.4s
//tensorflow/python/framework:error_interpolation_test                   PASSED in 10.7s
//tensorflow/python/framework:errors_test                                PASSED in 9.9s
//tensorflow/python/framework:extension_type_field_test                  PASSED in 24.9s
//tensorflow/python/framework:extension_type_test                        PASSED in 22.2s
//tensorflow/python/framework:file_system_test                           PASSED in 10.8s
//tensorflow/python/framework:flexible_dtypes_test                       PASSED in 122.0s
//tensorflow/python/framework:function_def_to_graph_test                 PASSED in 11.4s
//tensorflow/python/framework:graph_building_benchmark_cpu               PASSED in 13.9s
//tensorflow/python/framework:graph_util_test                            PASSED in 12.3s
//tensorflow/python/framework:immutable_dict_test                        PASSED in 10.1s
//tensorflow/python/framework:importer_test                              PASSED in 12.5s
//tensorflow/python/framework:indexed_slices_test                        PASSED in 16.2s
//tensorflow/python/framework:kernels_test                               PASSED in 13.1s
//tensorflow/python/framework:meta_graph_test                            PASSED in 15.9s
//tensorflow/python/framework:node_file_writer_test_cpu                  PASSED in 11.3s
//tensorflow/python/framework:offset_counter_helper_test                 PASSED in 0.2s
//tensorflow/python/framework:op_allowlist_namespace_test                PASSED in 3.1s
//tensorflow/python/framework:op_callbacks_test_cpu                      PASSED in 17.2s
//tensorflow/python/framework:op_def_library_test                        PASSED in 12.9s
//tensorflow/python/framework:op_def_util_test                           PASSED in 11.1s
//tensorflow/python/framework:ops_enable_eager_test                      PASSED in 3.2s
//tensorflow/python/framework:ops_test                                   PASSED in 24.2s
//tensorflow/python/framework:proto_test                                 PASSED in 10.8s
//tensorflow/python/framework:py_context_manager_test                    PASSED in 10.0s
//tensorflow/python/framework:python_api_dispatcher_test                 PASSED in 30.4s
//tensorflow/python/framework:python_api_info_test                       PASSED in 10.4s
//tensorflow/python/framework:python_api_parameter_converter_test        PASSED in 10.7s
//tensorflow/python/framework:python_op_gen_annotation_test              PASSED in 7.5s
//tensorflow/python/framework:python_op_gen_annotator_test               PASSED in 0.1s
//tensorflow/python/framework:python_op_gen_test                         PASSED in 0.1s
//tensorflow/python/framework:python_tensor_converter_test               PASSED in 11.0s
//tensorflow/python/framework:random_seed_test                           PASSED in 34.3s
//tensorflow/python/framework:registry_test                              PASSED in 14.0s
//tensorflow/python/framework:smart_cond_test                            PASSED in 96.6s
//tensorflow/python/framework:sparse_tensor_test                         PASSED in 11.6s
//tensorflow/python/framework:subscribe_test                             PASSED in 33.5s
//tensorflow/python/framework:tensor_shape_test                          PASSED in 10.6s
//tensorflow/python/framework:tensor_test                                PASSED in 10.0s
//tensorflow/python/framework:tensor_util_test                           PASSED in 13.6s
//tensorflow/python/framework:test_combinations_test                     PASSED in 24.5s
//tensorflow/python/framework:test_util_test_cpu                         PASSED in 20.1s
//tensorflow/python/framework:tf2_test                                   PASSED in 16.1s
//tensorflow/python/framework:traceable_stack_test                       PASSED in 12.6s
//tensorflow/python/framework:type_spec_test                             PASSED in 9.8s
//tensorflow/python/framework:versions_test                              PASSED in 11.9s
//tensorflow/python/framework:weak_tensor_test                           PASSED in 14.5s
//tensorflow/python/framework/experimental:graph_building_test_cpu       PASSED in 13.1s
//tensorflow/python/framework/experimental:unified_api_test_cpu          PASSED in 12.8s
//tensorflow/python/grappler:arithmetic_optimizer_test_cpu               PASSED in 10.5s
//tensorflow/python/grappler:auto_mixed_precision_test_cpu               PASSED in 14.4s
//tensorflow/python/grappler:constant_folding_test_cpu                   PASSED in 12.1s
//tensorflow/python/grappler:cost_analyzer_test                          PASSED in 13.2s
//tensorflow/python/grappler:datasets_test                               PASSED in 13.2s
//tensorflow/python/grappler:item_test                                   PASSED in 11.9s
//tensorflow/python/grappler:memory_optimizer_test                       PASSED in 20.7s
//tensorflow/python/grappler:model_analyzer_test                         PASSED in 15.2s
//tensorflow/python/grappler:remapper_test_cpu                           PASSED in 11.5s
//tensorflow/python/grappler:tf_optimizer_test                           PASSED in 11.0s
//tensorflow/python/kernel_tests:benchmark_test_cpu                      PASSED in 12.1s
//tensorflow/python/kernel_tests:check_ops_test_cpu                      PASSED in 24.7s
//tensorflow/python/kernel_tests:collective_ops_multi_worker_test        PASSED in 33.0s
//tensorflow/python/kernel_tests:composite_tensor_ops_test               PASSED in 14.7s
//tensorflow/python/kernel_tests:critical_section_test_cpu               PASSED in 48.6s
//tensorflow/python/kernel_tests:garbage_collection_test                 PASSED in 14.8s
//tensorflow/python/kernel_tests:gradient_correctness_test_cpu           PASSED in 14.0s
//tensorflow/python/kernel_tests:histogram_ops_test_cpu                  PASSED in 24.3s
//tensorflow/python/kernel_tests:logging_ops_test_cpu                    PASSED in 12.3s
//tensorflow/python/kernel_tests:numerics_test_cpu                       PASSED in 10.3s
//tensorflow/python/kernel_tests:template_test                           PASSED in 14.6s
//tensorflow/python/kernel_tests:trace_op_test_cpu                       PASSED in 17.6s
//tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu      PASSED in 13.3s
//tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test        PASSED in 17.1s
//tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu      PASSED in 17.4s
//tensorflow/python/kernel_tests/array_ops:bcast_ops_test                PASSED in 14.1s
//tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu           PASSED in 12.6s
//tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu     PASSED in 38.8s
//tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu              PASSED in 28.1s
//tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu    PASSED in 12.0s
//tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu          PASSED in 12.8s
//tensorflow/python/kernel_tests/array_ops:denormal_test_cpu             PASSED in 9.9s
//tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu      PASSED in 12.4s
//tensorflow/python/kernel_tests/array_ops:edit_distance_op_test         PASSED in 13.4s
//tensorflow/python/kernel_tests/array_ops:fingerprint_op_test           PASSED in 11.2s
//tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu         PASSED in 27.9s
//tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test         PASSED in 10.8s
//tensorflow/python/kernel_tests/array_ops:identity_op_py_test           PASSED in 12.5s
//tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu      PASSED in 15.9s
//tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu            PASSED in 12.7s
//tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu           PASSED in 18.0s
//tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu               PASSED in 30.4s
//tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu           PASSED in 17.8s
//tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu  PASSED in 13.7s
//tensorflow/python/kernel_tests/array_ops:scalar_test_cpu               PASSED in 15.1s
//tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu            PASSED in 16.3s
//tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu             PASSED in 12.4s
//tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu      PASSED in 17.8s
//tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu      PASSED in 13.2s
//tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu             PASSED in 22.3s
//tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu            PASSED in 11.1s
//tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu           PASSED in 13.5s
//tensorflow/python/kernel_tests/array_ops:where_op_test_cpu             PASSED in 17.3s
//tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu           PASSED in 75.2s
//tensorflow/python/kernel_tests/control_flow:control_flow_util_test     PASSED in 19.7s
//tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test  PASSED in 11.5s
//tensorflow/python/kernel_tests/control_flow:py_func_test_cpu           PASSED in 25.3s
//tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu          PASSED in 66.6s
//tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu          PASSED in 90.5s
//tensorflow/python/kernel_tests/custom_ops:ackermann_test               PASSED in 11.9s
//tensorflow/python/kernel_tests/custom_ops:duplicate_op_test            PASSED in 12.1s
//tensorflow/python/kernel_tests/custom_ops:invalid_op_test              PASSED in 12.4s
//tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 12.2s
//tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 15.5s
//tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 16.0s
//tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 11.4s
//tensorflow/python/kernel_tests/data_structures:fifo_queue_test         PASSED in 43.4s
//tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu       PASSED in 59.3s
//tensorflow/python/kernel_tests/data_structures:listdiff_op_test        PASSED in 12.4s
//tensorflow/python/kernel_tests/data_structures:lookup_ops_test         PASSED in 35.7s
//tensorflow/python/kernel_tests/data_structures:map_ops_test            PASSED in 16.3s
//tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 10.0s
//tensorflow/python/kernel_tests/data_structures:priority_queue_test     PASSED in 11.3s
//tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu      PASSED in 15.2s
//tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu       PASSED in 13.3s
//tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu        PASSED in 16.4s
//tensorflow/python/kernel_tests/distributions:bijector_test_cpu         PASSED in 12.1s
//tensorflow/python/kernel_tests/distributions:categorical_test_cpu      PASSED in 14.6s
//tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 17.3s
//tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu        PASSED in 17.1s
//tensorflow/python/kernel_tests/distributions:exponential_test_cpu      PASSED in 21.1s
//tensorflow/python/kernel_tests/distributions:gamma_test_cpu            PASSED in 51.6s
//tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 11.3s
//tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 12.5s
//tensorflow/python/kernel_tests/distributions:laplace_test_cpu          PASSED in 43.6s
//tensorflow/python/kernel_tests/distributions:multinomial_test_cpu      PASSED in 14.5s
//tensorflow/python/kernel_tests/distributions:normal_test_cpu           PASSED in 28.2s
//tensorflow/python/kernel_tests/distributions:special_math_test_cpu     PASSED in 28.4s
//tensorflow/python/kernel_tests/distributions:uniform_test_cpu          PASSED in 17.2s
//tensorflow/python/kernel_tests/image_ops:attention_ops_test            PASSED in 13.1s
//tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test            PASSED in 11.1s
//tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test     PASSED in 14.2s
//tensorflow/python/kernel_tests/image_ops:decode_image_op_test          PASSED in 10.9s
//tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test           PASSED in 10.9s
//tensorflow/python/kernel_tests/image_ops:decode_png_op_test            PASSED in 9.6s
//tensorflow/python/kernel_tests/image_ops:decode_raw_op_test            PASSED in 11.3s
//tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 11.3s
//tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 12.4s
//tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 12.6s
//tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test              PASSED in 22.5s
//tensorflow/python/kernel_tests/io_ops:decode_csv_op_test               PASSED in 10.8s
//tensorflow/python/kernel_tests/io_ops:io_ops_test                      PASSED in 10.9s
//tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test     PASSED in 12.3s
//tensorflow/python/kernel_tests/io_ops:parsing_ops_test                 PASSED in 30.1s
//tensorflow/python/kernel_tests/io_ops:reader_ops_test                  PASSED in 13.0s
//tensorflow/python/kernel_tests/io_ops:record_input_test                PASSED in 50.1s
//tensorflow/python/kernel_tests/io_ops:save_restore_ops_test            PASSED in 13.2s
//tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu          PASSED in 21.2s
//tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 17.9s
//tensorflow/python/kernel_tests/linalg:linear_operator_algebra_test_cpu PASSED in 11.5s
//tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu         PASSED in 14.4s
//tensorflow/python/kernel_tests/linalg:lu_op_test_cpu                   PASSED in 14.6s
//tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu       PASSED in 12.9s
//tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test         PASSED in 69.8s
//tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu      PASSED in 18.7s
//tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu         PASSED in 18.1s
//tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu   PASSED in 14.2s
//tensorflow/python/kernel_tests/linalg:slicing_test_cpu                 PASSED in 15.2s
//tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 20.3s
//tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 10.1s
//tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu         PASSED in 12.6s
//tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu             PASSED in 13.4s
//tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 13.4s
//tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu             PASSED in 11.5s
//tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu           PASSED in 14.1s
//tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu          PASSED in 21.5s
//tensorflow/python/kernel_tests/math_ops:clip_ops_test                  PASSED in 12.3s
//tensorflow/python/kernel_tests/math_ops:confusion_matrix_test          PASSED in 14.3s
//tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu            PASSED in 10.5s
//tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu  PASSED in 13.0s
//tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu            PASSED in 10.4s
//tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu      PASSED in 9.8s
//tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 12.5s
//tensorflow/python/kernel_tests/math_ops:sets_test                      PASSED in 57.9s
//tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu               PASSED in 33.9s
//tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu         PASSED in 14.5s
//tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu              PASSED in 13.5s
//tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu                 PASSED in 192.6s
//tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu                  PASSED in 13.0s
//tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu        PASSED in 11.4s
//tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu        PASSED in 12.1s
//tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 45.0s
//tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu        PASSED in 26.6s
//tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test             PASSED in 21.5s
//tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu             PASSED in 66.6s
//tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu               PASSED in 11.0s
//tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 10.9s
//tensorflow/python/kernel_tests/nn_ops:losses_test                      PASSED in 42.4s
//tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu                  PASSED in 13.0s
//tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu       PASSED in 31.7s
//tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu          PASSED in 18.1s
//tensorflow/python/kernel_tests/nn_ops:pool_test_cpu                    PASSED in 41.1s
//tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu          PASSED in 36.5s
//tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu                 PASSED in 21.6s
//tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu              PASSED in 11.6s
//tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu             PASSED in 10.2s
//tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu             PASSED in 11.4s
//tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu             PASSED in 131.1s
//tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu                 PASSED in 15.5s
//tensorflow/python/kernel_tests/proto:descriptor_source_test            PASSED in 13.7s
//tensorflow/python/kernel_tests/proto:encode_proto_op_test              PASSED in 20.0s
//tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test  PASSED in 17.2s
//tensorflow/python/kernel_tests/random:candidate_sampler_ops_test       PASSED in 11.7s
//tensorflow/python/kernel_tests/random:multinomial_op_test_cpu          PASSED in 10.5s
//tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 20.1s
//tensorflow/python/kernel_tests/random:random_crop_test_cpu             PASSED in 13.5s
//tensorflow/python/kernel_tests/random:random_grad_test_cpu             PASSED in 13.1s
//tensorflow/python/kernel_tests/random:random_ops_test_cpu              PASSED in 32.4s
//tensorflow/python/kernel_tests/random:random_poisson_test_cpu          PASSED in 41.8s
//tensorflow/python/kernel_tests/random:random_shuffle_queue_test        PASSED in 26.2s
//tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu     PASSED in 20.6s
//tensorflow/python/kernel_tests/signal:mel_ops_test_cpu                 PASSED in 33.1s
//tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu                PASSED in 96.3s
//tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu      PASSED in 42.5s
//tensorflow/python/kernel_tests/signal:shape_ops_test_cpu               PASSED in 22.9s
//tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test           PASSED in 14.3s
//tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test        PASSED in 11.3s
//tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 10.8s
//tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test         PASSED in 32.0s
//tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu    PASSED in 59.5s
//tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test       PASSED in 11.5s
//tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test       PASSED in 11.3s
//tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 15.2s
//tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test         PASSED in 23.5s
//tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu     PASSED in 13.1s
//tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 20.3s
//tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 41.4s
//tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 35.2s
//tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test  PASSED in 11.5s
//tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 11.4s
//tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu  PASSED in 78.8s
//tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu      PASSED in 12.1s
//tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test           PASSED in 11.5s
//tensorflow/python/kernel_tests/strings_ops:as_string_op_test           PASSED in 11.1s
//tensorflow/python/kernel_tests/strings_ops:base64_ops_test             PASSED in 22.0s
//tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu     PASSED in 14.5s
//tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test    PASSED in 25.8s
//tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test       PASSED in 9.6s
//tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test  PASSED in 10.4s
//tensorflow/python/kernel_tests/strings_ops:string_format_op_test       PASSED in 12.6s
//tensorflow/python/kernel_tests/strings_ops:string_join_op_test         PASSED in 12.6s
//tensorflow/python/kernel_tests/strings_ops:string_length_op_test       PASSED in 14.2s
//tensorflow/python/kernel_tests/strings_ops:string_lower_op_test        PASSED in 11.9s
//tensorflow/python/kernel_tests/strings_ops:string_split_op_test        PASSED in 16.3s
//tensorflow/python/kernel_tests/strings_ops:string_strip_op_test        PASSED in 14.2s
//tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 11.7s
//tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 11.4s
//tensorflow/python/kernel_tests/strings_ops:string_upper_op_test        PASSED in 13.6s
//tensorflow/python/kernel_tests/strings_ops:substr_op_test              PASSED in 14.6s
//tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test      PASSED in 19.6s
//tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test      PASSED in 10.8s
//tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test      PASSED in 13.3s
//tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test   PASSED in 10.8s
//tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 11.3s
//tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu        PASSED in 22.2s
//tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 17.8s
//tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 28.8s
//tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test         PASSED in 11.2s
//tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test   PASSED in 10.9s
//tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu      PASSED in 12.7s
//tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 10.3s
//tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test     PASSED in 10.6s
//tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 13.7s
//tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu    PASSED in 10.3s
//tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu       PASSED in 9.5s
//tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 14.6s
//tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu     PASSED in 12.8s
//tensorflow/python/kernel_tests/variables:partitioned_variables_test    PASSED in 16.9s
//tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 112.7s
//tensorflow/python/kernel_tests/variables:variable_ops_test_cpu         PASSED in 14.4s
//tensorflow/python/kernel_tests/variables:variable_scope_test           PASSED in 37.8s
//tensorflow/python/kernel_tests/variables:variables_test                PASSED in 16.5s
//tensorflow/python/lib/io:file_io_test                                  PASSED in 15.8s
//tensorflow/python/lib/io:tf_record_test                                PASSED in 21.8s
//tensorflow/python/module:module_test                                   PASSED in 11.2s
//tensorflow/python/ops:array_grad_test_cpu                              PASSED in 13.3s
//tensorflow/python/ops:array_ops_shape_test                             PASSED in 10.6s
//tensorflow/python/ops:array_ops_test                                   PASSED in 11.1s
//tensorflow/python/ops:autograph_ops_test                               PASSED in 41.6s
//tensorflow/python/ops:batch_norm_benchmark_cpu                         PASSED in 10.6s
//tensorflow/python/ops:bincount_ops_test_cpu                            PASSED in 11.2s
//tensorflow/python/ops:bitwise_ops_test_cpu                             PASSED in 23.4s
//tensorflow/python/ops:clip_ops_test                                    PASSED in 90.4s
//tensorflow/python/ops:clustering_ops_test                              PASSED in 25.9s
//tensorflow/python/ops:collective_ops_benchmark_cpu                     PASSED in 10.3s
//tensorflow/python/ops:collective_ops_gpu_test_2gpu                     PASSED in 28.9s
//tensorflow/python/ops:collective_ops_gpu_test_cpu                      PASSED in 12.9s
//tensorflow/python/ops:collective_ops_test                              PASSED in 24.3s
//tensorflow/python/ops:collective_ops_xla_test                          PASSED in 14.0s
//tensorflow/python/ops:compiled_collective_ops_gpu_test_2gpu            PASSED in 14.9s
//tensorflow/python/ops:compiled_collective_ops_gpu_test_cpu             PASSED in 11.0s
//tensorflow/python/ops:concat_benchmark_cpu                             PASSED in 13.5s
//tensorflow/python/ops:control_flow_ops_benchmark_cpu                   PASSED in 18.3s
//tensorflow/python/ops:control_flow_v2_enable_test                      PASSED in 10.8s
//tensorflow/python/ops:control_flow_v2_toggles_test                     PASSED in 9.4s
//tensorflow/python/ops:dequantize_op_test                               PASSED in 12.4s
//tensorflow/python/ops:embedding_ops_test_cpu                           PASSED in 10.8s
//tensorflow/python/ops:factory_ops_test_cpu                             PASSED in 11.1s
//tensorflow/python/ops:functional_ops_test                              PASSED in 12.4s
//tensorflow/python/ops:gradient_checker_v2_test_cpu                     PASSED in 36.9s
//tensorflow/python/ops:gradients_test_cpu                               PASSED in 29.9s
//tensorflow/python/ops:init_ops_test_cpu                                PASSED in 12.4s
//tensorflow/python/ops:init_ops_v2_test_cpu                             PASSED in 43.8s
//tensorflow/python/ops:math_grad_test_cpu                               PASSED in 20.0s
//tensorflow/python/ops:math_ops_linspace_test_cpu                       PASSED in 15.2s
//tensorflow/python/ops:math_ops_test_cpu                                PASSED in 54.8s
//tensorflow/python/ops:matmul_benchmark_cpu                             PASSED in 11.8s
//tensorflow/python/ops:nn_grad_test_cpu                                 PASSED in 13.8s
//tensorflow/python/ops:nn_loss_scaling_utilities_test                   PASSED in 15.4s
//tensorflow/python/ops:nn_test_cpu                                      PASSED in 72.4s
//tensorflow/python/ops:nn_xent_test_cpu                                 PASSED in 11.0s
//tensorflow/python/ops:op_selector_test                                 PASSED in 10.1s
//tensorflow/python/ops:quantized_conv_ops_test                          PASSED in 9.7s
//tensorflow/python/ops:quantized_ops_test                               PASSED in 10.7s
//tensorflow/python/ops:raw_ops_test_cpu                                 PASSED in 10.9s
//tensorflow/python/ops:rnn_grad_test_cpu                                PASSED in 13.5s
//tensorflow/python/ops:script_ops_test                                  PASSED in 10.0s
//tensorflow/python/ops:sort_ops_test                                    PASSED in 11.0s
//tensorflow/python/ops:sparse_bincount_ops_test_cpu                     PASSED in 17.0s
//tensorflow/python/ops:sparse_ops_test                                  PASSED in 19.4s
//tensorflow/python/ops:split_benchmark_cpu                              PASSED in 10.6s
//tensorflow/python/ops:tensor_array_ops_test                            PASSED in 10.3s
//tensorflow/python/ops:transpose_benchmark_cpu                          PASSED in 13.0s
//tensorflow/python/ops:variable_spec_test                               PASSED in 12.5s
//tensorflow/python/ops:weak_tensor_array_ops_test                       PASSED in 10.5s
//tensorflow/python/ops:weak_tensor_image_ops_test                       PASSED in 9.4s
//tensorflow/python/ops:weak_tensor_math_ops_test                        PASSED in 44.7s
//tensorflow/python/ops:weak_tensor_nn_test_cpu                          PASSED in 33.8s
//tensorflow/python/ops:weak_tensor_np_array_ops_test                    PASSED in 41.0s
//tensorflow/python/ops:weak_tensor_np_math_ops_test                     PASSED in 12.4s
//tensorflow/python/ops:weak_tensor_ops_test                             PASSED in 74.7s
//tensorflow/python/ops/losses:util_test                                 PASSED in 9.9s
//tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu     PASSED in 15.5s
//tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu                  PASSED in 117.2s
//tensorflow/python/ops/numpy_ops:np_arrays_test_cpu                     PASSED in 11.6s
//tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu                     PASSED in 14.0s
//tensorflow/python/ops/numpy_ops:np_interop_test_cpu                    PASSED in 70.8s
//tensorflow/python/ops/numpy_ops:np_logic_test_cpu                      PASSED in 32.5s
//tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu                   PASSED in 34.5s
//tensorflow/python/ops/numpy_ops:np_random_test_cpu                     PASSED in 72.0s
//tensorflow/python/ops/numpy_ops:np_utils_test_cpu                      PASSED in 29.1s
//tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu    PASSED in 25.7s
//tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test    PASSED in 28.9s
//tensorflow/python/ops/parallel_for:array_test_cpu                      PASSED in 68.4s
//tensorflow/python/ops/parallel_for:gradients_test_cpu                  PASSED in 14.6s
//tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu       PASSED in 87.2s
//tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 15.4s
//tensorflow/python/ops/ragged:ragged_batch_gather_op_test               PASSED in 40.4s
//tensorflow/python/ops/ragged:ragged_bincount_ops_test_cpu              PASSED in 11.2s
//tensorflow/python/ops/ragged:ragged_bitcast_op_test                    PASSED in 9.3s
//tensorflow/python/ops/ragged:ragged_boolean_mask_op_test               PASSED in 20.0s
//tensorflow/python/ops/ragged:ragged_concat_op_test                     PASSED in 12.7s
//tensorflow/python/ops/ragged:ragged_const_op_test                      PASSED in 14.3s
//tensorflow/python/ops/ragged:ragged_constant_value_op_test             PASSED in 11.1s
//tensorflow/python/ops/ragged:ragged_cross_op_test                      PASSED in 48.2s
//tensorflow/python/ops/ragged:ragged_dispatch_test                      PASSED in 165.7s
//tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu      PASSED in 23.2s
//tensorflow/python/ops/ragged:ragged_eager_test                         PASSED in 14.8s
//tensorflow/python/ops/ragged:ragged_expand_dims_op_test                PASSED in 11.1s
//tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu               PASSED in 20.8s
//tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test            PASSED in 14.6s
//tensorflow/python/ops/ragged:ragged_from_sparse_op_test                PASSED in 14.5s
//tensorflow/python/ops/ragged:ragged_from_tensor_op_test                PASSED in 25.8s
//tensorflow/python/ops/ragged:ragged_gather_nd_op_test                  PASSED in 89.9s
//tensorflow/python/ops/ragged:ragged_map_flat_values_op_test            PASSED in 12.9s
//tensorflow/python/ops/ragged:ragged_map_fn_op_test                     PASSED in 19.9s
//tensorflow/python/ops/ragged:ragged_math_ops_test                      PASSED in 16.4s
//tensorflow/python/ops/ragged:ragged_matmul_op_test                     PASSED in 41.4s
//tensorflow/python/ops/ragged:ragged_merge_dims_op_test                 PASSED in 31.0s
//tensorflow/python/ops/ragged:ragged_one_hot_op_test                    PASSED in 42.6s
//tensorflow/python/ops/ragged:ragged_operators_test                     PASSED in 33.1s
//tensorflow/python/ops/ragged:ragged_placeholder_op_test                PASSED in 11.6s
//tensorflow/python/ops/ragged:ragged_print_op_test                      PASSED in 18.7s
//tensorflow/python/ops/ragged:ragged_range_op_test                      PASSED in 11.6s
//tensorflow/python/ops/ragged:ragged_rank_op_test                       PASSED in 40.1s
//tensorflow/python/ops/ragged:ragged_reduce_op_test                     PASSED in 39.1s
//tensorflow/python/ops/ragged:ragged_resize_image_op_test               PASSED in 22.2s
//tensorflow/python/ops/ragged:ragged_reverse_op_test                    PASSED in 18.9s
//tensorflow/python/ops/ragged:ragged_row_lengths_op_test                PASSED in 14.5s
//tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test  PASSED in 29.2s
//tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test  PASSED in 10.5s
//tensorflow/python/ops/ragged:ragged_segment_op_test                    PASSED in 19.3s
//tensorflow/python/ops/ragged:ragged_size_op_test                       PASSED in 10.4s
//tensorflow/python/ops/ragged:ragged_split_op_test                      PASSED in 44.5s
//tensorflow/python/ops/ragged:ragged_squeeze_op_test                    PASSED in 56.6s
//tensorflow/python/ops/ragged:ragged_stack_op_test                      PASSED in 46.2s
//tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test      PASSED in 15.6s
//tensorflow/python/ops/ragged:ragged_tensor_shape_test                  PASSED in 62.1s
//tensorflow/python/ops/ragged:ragged_tile_op_test                       PASSED in 59.4s
//tensorflow/python/ops/ragged:ragged_to_sparse_op_test                  PASSED in 17.5s
//tensorflow/python/ops/ragged:ragged_to_tensor_op_test                  PASSED in 75.7s
//tensorflow/python/ops/ragged:ragged_util_test                          PASSED in 28.7s
//tensorflow/python/ops/ragged:ragged_where_op_test                      PASSED in 47.3s
//tensorflow/python/ops/ragged:row_partition_test                        PASSED in 36.0s
//tensorflow/python/ops/ragged:string_ngrams_op_test                     PASSED in 11.9s
//tensorflow/python/ops/ragged:strings_reduce_join_op_test               PASSED in 14.3s
//tensorflow/python/ops/structured:structured_array_ops_test             PASSED in 47.4s
//tensorflow/python/ops/structured:structured_tensor_slice_test          PASSED in 65.5s
//tensorflow/python/ops/structured:structured_tensor_spec_test           PASSED in 13.0s
//tensorflow/python/ops/structured:structured_tensor_test                PASSED in 62.2s
//tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu        PASSED in 17.7s
//tensorflow/python/platform:benchmark_test                              PASSED in 15.0s
//tensorflow/python/platform:build_info_test                             PASSED in 12.2s
//tensorflow/python/platform:resource_loader_test                        PASSED in 4.9s
//tensorflow/python/profiler:pprof_profiler_test                         PASSED in 25.3s
//tensorflow/python/profiler:profile_context_test_cpu                    PASSED in 25.9s
//tensorflow/python/profiler:profiler_client_test_cpu                    PASSED in 11.4s
//tensorflow/python/profiler:profiler_test_cpu                           PASSED in 21.1s
//tensorflow/python/profiler:profiler_v2_test_cpu                        PASSED in 9.8s
//tensorflow/python/profiler:profiler_wrapper_test                       PASSED in 9.0s
//tensorflow/python/profiler:tfprof_logger_test                          PASSED in 15.7s
//tensorflow/python/profiler/internal:flops_registry_test                PASSED in 11.4s
//tensorflow/python/profiler/internal:print_model_analysis_test          PASSED in 11.0s
//tensorflow/python/profiler/internal:run_metadata_test_cpu              PASSED in 17.4s
//tensorflow/python/saved_model:fingerprinting_test                      PASSED in 14.5s
//tensorflow/python/saved_model:keras_injection_test                     PASSED in 54.9s
//tensorflow/python/saved_model:load_v1_in_v2_test                       PASSED in 34.8s
//tensorflow/python/saved_model:loader_test                              PASSED in 15.1s
//tensorflow/python/saved_model:method_name_updater_test                 PASSED in 15.8s
//tensorflow/python/saved_model:metrics_test                             PASSED in 13.6s
//tensorflow/python/saved_model:nested_structure_coder_test              PASSED in 10.4s
//tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test   PASSED in 16.1s
//tensorflow/python/saved_model:pywrap_saved_model_metrics_test          PASSED in 12.0s
//tensorflow/python/saved_model:revived_types_test                       PASSED in 10.9s
//tensorflow/python/saved_model:save_context_test                        PASSED in 10.7s
//tensorflow/python/saved_model:save_test                                PASSED in 31.7s
//tensorflow/python/saved_model:saved_model_test                         PASSED in 55.1s
//tensorflow/python/saved_model:signature_def_utils_test                 PASSED in 13.7s
//tensorflow/python/saved_model:simple_save_test                         PASSED in 10.1s
//tensorflow/python/saved_model:tracing_utils_test                       PASSED in 10.8s
//tensorflow/python/saved_model:utils_test                               PASSED in 13.9s
//tensorflow/python/saved_model/model_utils:export_output_test           PASSED in 12.0s
//tensorflow/python/saved_model/model_utils:export_test                  PASSED in 14.2s
//tensorflow/python/saved_model/model_utils:mode_keys_test               PASSED in 10.4s
//tensorflow/python/saved_model/registration:registration_saving_test    PASSED in 21.8s
//tensorflow/python/saved_model/registration:registration_test           PASSED in 10.1s
//tensorflow/python/saved_model/registration:tf_registration_test        PASSED in 22.5s
//tensorflow/python/saved_model/tests:variable_wrapper_test              PASSED in 11.3s
//tensorflow/python/summary:plugin_asset_test                            PASSED in 11.6s
//tensorflow/python/summary:summary_iterator_test                        PASSED in 9.8s
//tensorflow/python/summary:summary_test                                 PASSED in 10.8s
//tensorflow/python/summary:summary_v2_test                              PASSED in 12.6s
//tensorflow/python/summary/writer:writer_test                           PASSED in 21.7s
//tensorflow/python/tools:aot_compiled_test                              PASSED in 19.4s
//tensorflow/python/tools:freeze_graph_test                              PASSED in 17.9s
//tensorflow/python/tools:optimize_for_inference_test                    PASSED in 10.2s
//tensorflow/python/tools:print_selective_registration_header_test       PASSED in 23.4s
//tensorflow/python/tools:saved_model_cli_test                           PASSED in 31.7s
//tensorflow/python/tools:saved_model_utils_test                         PASSED in 10.9s
//tensorflow/python/tools:strip_unused_test                              PASSED in 10.1s
//tensorflow/python/tools/api/generator:create_python_api_test           PASSED in 11.2s
//tensorflow/python/tools/api/generator:output_init_files_test           PASSED in 22.4s
//tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test         PASSED in 14.9s
//tensorflow/python/tpu:bfloat16_test                                    PASSED in 10.9s
//tensorflow/python/tpu:feature_column_test                              PASSED in 44.6s
//tensorflow/python/tpu:topology_test                                    PASSED in 10.5s
//tensorflow/python/tpu:tpu_embedding_for_serving_test                   PASSED in 14.1s
//tensorflow/python/tpu:tpu_embedding_v2_utils_test                      PASSED in 10.3s
//tensorflow/python/tpu:tpu_infeed_test                                  PASSED in 13.1s
//tensorflow/python/tpu:tpu_sharding_test                                PASSED in 10.5s
//tensorflow/python/tpu:tpu_test_wrapper_test                            PASSED in 14.2s
//tensorflow/python/tpu/client:client_py_test                            PASSED in 10.5s
//tensorflow/python/trackable:autotrackable_test                         PASSED in 11.9s
//tensorflow/python/trackable:base_delegate_test                         PASSED in 35.3s
//tensorflow/python/trackable:base_test                                  PASSED in 40.2s
//tensorflow/python/trackable:data_structures_test                       PASSED in 18.4s
//tensorflow/python/trackable:python_state_test                          PASSED in 12.1s
//tensorflow/python/trackable:resource_test                              PASSED in 13.7s
//tensorflow/python/trackable:trackable_utils_test                       PASSED in 12.1s
//tensorflow/python/training:adadelta_test_cpu                           PASSED in 20.8s
//tensorflow/python/training:adagrad_da_test_cpu                         PASSED in 14.0s
//tensorflow/python/training:adagrad_test_cpu                            PASSED in 14.8s
//tensorflow/python/training:adam_test_cpu                               PASSED in 21.9s
//tensorflow/python/training:basic_loops_test_cpu                        PASSED in 12.6s
//tensorflow/python/training:basic_session_run_hooks_test                PASSED in 28.0s
//tensorflow/python/training:checkpoint_ops_test                         PASSED in 12.1s
//tensorflow/python/training:coordinator_test_cpu                        PASSED in 97.7s
//tensorflow/python/training:device_setter_test_cpu                      PASSED in 11.3s
//tensorflow/python/training:ftrl_test_cpu                               PASSED in 19.0s
//tensorflow/python/training:gradient_descent_test_cpu                   PASSED in 14.5s
//tensorflow/python/training:input_test                                  PASSED in 50.1s
//tensorflow/python/training:momentum_test_cpu                           PASSED in 17.2s
//tensorflow/python/training:monitored_session_test                      PASSED in 29.9s
//tensorflow/python/training:moving_averages_test_cpu                    PASSED in 26.3s
//tensorflow/python/training:optimizer_test_cpu                          PASSED in 35.7s
//tensorflow/python/training:proximal_adagrad_test_cpu                   PASSED in 12.4s
//tensorflow/python/training:proximal_gradient_descent_test_cpu          PASSED in 12.4s
//tensorflow/python/training:quantize_training_test_cpu                  PASSED in 10.2s
//tensorflow/python/training:queue_runner_test_cpu                       PASSED in 13.1s
//tensorflow/python/training:rmsprop_test_cpu                            PASSED in 31.2s
//tensorflow/python/training:saver_large_partitioned_variable_test       PASSED in 20.1s
//tensorflow/python/training:saver_test_2gpu                             PASSED in 45.0s
//tensorflow/python/training:saver_test_cpu                              PASSED in 44.0s
//tensorflow/python/training:server_lib_multiple_containers_test         PASSED in 12.3s
//tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 11.3s
//tensorflow/python/training:server_lib_same_variables_clear_test        PASSED in 12.2s
//tensorflow/python/training:server_lib_same_variables_no_clear_test     PASSED in 9.9s
//tensorflow/python/training:server_lib_sparse_job_test                  PASSED in 27.5s
//tensorflow/python/training:server_lib_test                             PASSED in 34.1s
//tensorflow/python/training:session_manager_test_cpu                    PASSED in 108.2s
//tensorflow/python/training:slot_creator_test_cpu                       PASSED in 11.2s
//tensorflow/python/training:supervisor_test                             PASSED in 18.3s
//tensorflow/python/training:training_ops_mlir_test_cpu                  PASSED in 11.4s
//tensorflow/python/training:training_ops_test_cpu                       PASSED in 14.0s
//tensorflow/python/training:training_util_test                          PASSED in 14.1s
//tensorflow/python/training:warm_starting_util_test                     PASSED in 30.9s
//tensorflow/python/training/experimental:loss_scale_optimizer_test      PASSED in 19.2s
//tensorflow/python/training/experimental:loss_scale_test                PASSED in 29.3s
//tensorflow/python/training/experimental:mixed_precision_test_cpu       PASSED in 11.1s
//tensorflow/python/training/saving:saveable_object_util_test            PASSED in 13.6s
//tensorflow/python/util:compat_test                                     PASSED in 9.4s
//tensorflow/python/util:decorator_utils_test                            PASSED in 14.3s
//tensorflow/python/util:deprecation_test                                PASSED in 15.2s
//tensorflow/python/util:dispatch_test                                   PASSED in 13.2s
//tensorflow/python/util:example_parser_configuration_test               PASSED in 11.0s
//tensorflow/python/util:fast_module_type_test                           PASSED in 11.9s
//tensorflow/python/util:function_parameter_canonicalizer_test           PASSED in 11.4s
//tensorflow/python/util:function_utils_test                             PASSED in 12.1s
//tensorflow/python/util:keyword_args_test                               PASSED in 13.2s
//tensorflow/python/util:lazy_loader_test                                PASSED in 19.4s
//tensorflow/python/util:lock_util_test                                  PASSED in 14.8s
//tensorflow/python/util:module_wrapper_test                             PASSED in 12.3s
//tensorflow/python/util:nest_test                                       PASSED in 33.7s
//tensorflow/python/util:object_identity_test                            PASSED in 10.0s
//tensorflow/python/util:pywrap_xla_ops_test                             PASSED in 5.5s
//tensorflow/python/util:serialization_test                              PASSED in 11.1s
//tensorflow/python/util:tf_contextlib_test                              PASSED in 10.8s
//tensorflow/python/util:tf_decorator_test                               PASSED in 30.0s
//tensorflow/python/util:tf_export_test                                  PASSED in 17.6s
//tensorflow/python/util:tf_inspect_test                                 PASSED in 41.3s
//tensorflow/python/util:tf_should_use_test                              PASSED in 17.4s
//tensorflow/python/util:tf_stack_test                                   PASSED in 11.2s
//tensorflow/python/util:traceback_utils_test                            PASSED in 41.4s
//tensorflow/python/util:type_annotations_test                           PASSED in 12.5s
//tensorflow/python/util:variable_utils_test                             PASSED in 66.7s
//tensorflow/python/util:vlog_test                                       PASSED in 10.5s
//tensorflow/tools/api/tests:module_test                                 PASSED in 22.9s
//tensorflow/tools/benchmark:benchmark_model_test                        PASSED in 3.5s
//tensorflow/tools/common:public_api_test                                PASSED in 3.0s
//tensorflow/tools/common:traverse_test                                  PASSED in 3.6s
//tensorflow/tools/compatibility:all_renames_v2_test                     PASSED in 10.2s
//tensorflow/tools/compatibility:ast_edits_test                          PASSED in 9.6s
//tensorflow/tools/compatibility:test_file_v1_0                          PASSED in 20.2s
//tensorflow/tools/compatibility:test_file_v2_0                          PASSED in 57.9s
//tensorflow/tools/compatibility:tf_upgrade_test                         PASSED in 10.1s
//tensorflow/tools/compatibility:tf_upgrade_v2_safety_test               PASSED in 9.6s
//tensorflow/tools/docs:tf_doctest_test                                  PASSED in 1.6s
//tensorflow/tools/graph_transforms:file_utils_test                      PASSED in 5.8s
//tensorflow/tools/graph_transforms:transform_graph_test                 PASSED in 2.6s
//tensorflow/tools/graph_transforms:transform_utils_test                 PASSED in 2.6s
//tensorflow/tools/graph_transforms:transforms_test                      PASSED in 4.1s
//tensorflow/tools/proto_splitter:merge_test                             PASSED in 0.2s
//tensorflow/tools/proto_splitter:split_graph_def_test                   PASSED in 9.1s
//tensorflow/tools/proto_splitter:split_test                             PASSED in 16.4s
//tensorflow/tools/proto_splitter:util_test                              PASSED in 10.7s
//tensorflow/tools/proto_splitter/cc:composable_splitter_test            PASSED in 0.5s
//tensorflow/tools/proto_splitter/cc:graph_def_splitter_test             PASSED in 0.2s
//tensorflow/tools/proto_splitter/cc:saved_model_splitter_test           PASSED in 1.4s
//tensorflow/tools/proto_splitter/cc:util_test                           PASSED in 2.7s
//tensorflow/tools/proto_splitter/python:saved_model_test                PASSED in 10.4s
//tensorflow/tools/proto_splitter/python:test_util_test                  PASSED in 11.1s
//tensorflow/tools/proto_text:gen_proto_text_functions_lib_test          PASSED in 0.8s
//tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.4s
//tensorflow/tsl/c:tsl_status_test                                       PASSED in 0.1s
//tensorflow/tsl/concurrency:async_value_ref_test                        PASSED in 0.7s
//tensorflow/tsl/concurrency:async_value_test                            PASSED in 0.1s
//tensorflow/tsl/concurrency:concurrent_vector_test                      PASSED in 0.2s
//tensorflow/tsl/cuda:cudnn_version_test                                 PASSED in 0.1s
//tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 13.1s
//tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.1s
//tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 0.2s
//tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 5.2s
//tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 5.4s
//tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test               PASSED in 0.2s
//tensorflow/tsl/distributed_runtime/rpc:grpc_util_test                  PASSED in 0.1s
//tensorflow/tsl/framework:cancellation_test                             PASSED in 1.4s
//tensorflow/tsl/framework:device_id_utils_test                          PASSED in 3.1s
//tensorflow/tsl/framework/convolution:eigen_spatial_convolutions_test   PASSED in 0.1s
//tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests                               PASSED in 0.1s
//tensorflow/tsl/lib/hash:crc32c_test                                    PASSED in 0.1s
//tensorflow/tsl/lib/histogram:histogram_test                            PASSED in 0.1s
//tensorflow/tsl/lib/io:buffered_file_test                               PASSED in 0.2s
//tensorflow/tsl/lib/io:buffered_inputstream_test                        PASSED in 0.3s
//tensorflow/tsl/lib/io:cache_test                                       PASSED in 0.3s
//tensorflow/tsl/lib/io:inputbuffer_test                                 PASSED in 1.1s
//tensorflow/tsl/lib/io:inputstream_interface_test                       PASSED in 0.1s
//tensorflow/tsl/lib/io:random_inputstream_test                          PASSED in 0.2s
//tensorflow/tsl/lib/io:record_reader_writer_test                        PASSED in 1.6s
//tensorflow/tsl/lib/io:recordio_test                                    PASSED in 0.7s
//tensorflow/tsl/lib/io:table_test                                       PASSED in 4.1s
//tensorflow/tsl/lib/io:zlib_buffers_test                                PASSED in 6.2s
//tensorflow/tsl/lib/io/snappy:snappy_test                               PASSED in 1.6s
//tensorflow/tsl/lib/math:math_util_test                                 PASSED in 0.1s
//tensorflow/tsl/lib/random:distribution_sampler_test                    PASSED in 0.3s
//tensorflow/tsl/lib/random:philox_random_test                           PASSED in 0.1s
//tensorflow/tsl/lib/random:random_distributions_test                    PASSED in 19.7s
//tensorflow/tsl/lib/random:simple_philox_test                           PASSED in 0.3s
//tensorflow/tsl/lib/random:weighted_picker_test                         PASSED in 10.0s
//tensorflow/tsl/platform:criticality_test                               PASSED in 0.1s
//tensorflow/tsl/platform:ctstring_test                                  PASSED in 0.1s
//tensorflow/tsl/platform:denormal_test                                  PASSED in 0.1s
//tensorflow/tsl/platform:errors_test                                    PASSED in 0.2s
//tensorflow/tsl/platform:fingerprint_test                               PASSED in 0.6s
//tensorflow/tsl/platform:hash_test                                      PASSED in 0.2s
//tensorflow/tsl/platform:integral_types_test                            PASSED in 0.1s
//tensorflow/tsl/platform:intrusive_ptr_test                             PASSED in 0.1s
//tensorflow/tsl/platform:logging_test                                   PASSED in 20.0s
//tensorflow/tsl/platform:mutex_test                                     PASSED in 0.2s
//tensorflow/tsl/platform:net_test                                       PASSED in 0.2s
//tensorflow/tsl/platform:numbers_test                                   PASSED in 0.1s
//tensorflow/tsl/platform:path_test                                      PASSED in 0.1s
//tensorflow/tsl/platform:port_test                                      PASSED in 8.6s
//tensorflow/tsl/platform:random_test                                    PASSED in 3.1s
//tensorflow/tsl/platform:refcount_test                                  PASSED in 0.2s
//tensorflow/tsl/platform:retrying_file_system_test                      PASSED in 0.4s
//tensorflow/tsl/platform:retrying_utils_test                            PASSED in 0.1s
//tensorflow/tsl/platform:scanner_test                                   PASSED in 0.1s
//tensorflow/tsl/platform:setround_test                                  PASSED in 0.1s
//tensorflow/tsl/platform:stacktrace_handler_test                        PASSED in 1.9s
//tensorflow/tsl/platform:stacktrace_test                                PASSED in 0.4s
//tensorflow/tsl/platform:status_matchers_test                           PASSED in 0.1s
//tensorflow/tsl/platform:status_test                                    PASSED in 0.1s
//tensorflow/tsl/platform:statusor_test                                  PASSED in 1.6s
//tensorflow/tsl/platform:str_util_test                                  PASSED in 0.1s
//tensorflow/tsl/platform:strcat_test                                    PASSED in 0.9s
//tensorflow/tsl/platform:stringpiece_test                               PASSED in 0.1s
//tensorflow/tsl/platform:stringprintf_test                              PASSED in 0.3s
//tensorflow/tsl/platform:subprocess_test                                PASSED in 0.1s
//tensorflow/tsl/platform:tstring_test                                   PASSED in 0.1s
//tensorflow/tsl/platform:unbounded_work_queue_test                      PASSED in 0.2s
//tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test      PASSED in 0.1s
//tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test        PASSED in 0.1s
//tensorflow/tsl/platform/cloud:curl_http_request_test                   PASSED in 6.4s
//tensorflow/tsl/platform/cloud:expiring_lru_cache_test                  PASSED in 0.3s
//tensorflow/tsl/platform/cloud:gcs_dns_cache_test                       PASSED in 2.4s
//tensorflow/tsl/platform/cloud:gcs_file_system_test                     PASSED in 11.8s
//tensorflow/tsl/platform/cloud:gcs_throttle_test                        PASSED in 0.2s
//tensorflow/tsl/platform/cloud:google_auth_provider_test                PASSED in 0.1s
//tensorflow/tsl/platform/cloud:oauth_client_test                        PASSED in 0.7s
//tensorflow/tsl/platform/cloud:ram_file_block_cache_test                PASSED in 2.6s
//tensorflow/tsl/platform/cloud:time_util_test                           PASSED in 0.1s
//tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test             PASSED in 0.1s
//tensorflow/tsl/profiler/convert:trace_container_test                   PASSED in 0.1s
//tensorflow/tsl/profiler/convert:trace_events_to_json_test              PASSED in 0.1s
//tensorflow/tsl/profiler/convert:xla_op_utils_test                      PASSED in 0.1s
//tensorflow/tsl/profiler/convert:xplane_to_trace_events_test            PASSED in 0.1s
//tensorflow/tsl/profiler/lib:profiler_factory_test                      PASSED in 0.2s
//tensorflow/tsl/profiler/lib:profiler_lock_test                         PASSED in 0.2s
//tensorflow/tsl/profiler/lib:scoped_annotation_test                     PASSED in 0.1s
//tensorflow/tsl/profiler/lib:traceme_encode_test                        PASSED in 0.1s
//tensorflow/tsl/profiler/rpc/client:profiler_client_test                PASSED in 3.5s
//tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.4s
//tensorflow/tsl/profiler/utils:buffer_pool_test                         PASSED in 0.1s
//tensorflow/tsl/profiler/utils:group_events_test                        PASSED in 0.2s
//tensorflow/tsl/profiler/utils:parse_annotation_test                    PASSED in 0.1s
//tensorflow/tsl/profiler/utils:preprocess_xplane_test                   PASSED in 0.1s
//tensorflow/tsl/profiler/utils:tf_op_utils_test                         PASSED in 0.2s
//tensorflow/tsl/profiler/utils:timespan_test                            PASSED in 0.1s
//tensorflow/tsl/profiler/utils:tpu_xplane_utils_test                    PASSED in 0.5s
//tensorflow/tsl/profiler/utils:xplane_builder_test                      PASSED in 0.1s
//tensorflow/tsl/profiler/utils:xplane_utils_test                        PASSED in 0.1s
//tensorflow/tsl/util:device_name_utils_test                             PASSED in 0.1s
//tensorflow/tsl/util:stats_calculator_test                              PASSED in 0.1s
//tensorflow/compiler/tests:complex_div_test_cpu                         PASSED in 8.7s
  Stats over 2 runs: max = 8.7s, min = 7.7s, avg = 8.2s, dev = 0.5s
//tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test        PASSED in 10.0s
  Stats over 2 runs: max = 10.0s, min = 9.2s, avg = 9.6s, dev = 0.4s
//tensorflow/compiler/xla/tests:conditional_test_cpu                     PASSED in 11.7s
  Stats over 2 runs: max = 11.7s, min = 10.3s, avg = 11.0s, dev = 0.7s
//tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 24.7s
  Stats over 2 runs: max = 24.7s, min = 17.1s, avg = 20.9s, dev = 3.8s
//tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 18.4s
  Stats over 2 runs: max = 18.4s, min = 16.5s, avg = 17.5s, dev = 0.9s
//tensorflow/python/data/kernel_tests:padded_batch_test                  PASSED in 41.4s
  Stats over 2 runs: max = 41.4s, min = 35.9s, avg = 38.6s, dev = 2.8s
//tensorflow/python/data/kernel_tests:repeat_test                        PASSED in 57.6s
  Stats over 2 runs: max = 57.6s, min = 55.6s, avg = 56.6s, dev = 1.0s
//tensorflow/python/data/kernel_tests:window_test                        PASSED in 37.7s
  Stats over 2 runs: max = 37.7s, min = 30.8s, avg = 34.2s, dev = 3.4s
//tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu       PASSED in 15.7s
  Stats over 2 runs: max = 15.7s, min = 15.7s, avg = 15.7s, dev = 0.0s
//tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu    PASSED in 34.3s
  Stats over 2 runs: max = 34.3s, min = 33.5s, avg = 33.9s, dev = 0.4s
//tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu            PASSED in 13.2s
  Stats over 2 runs: max = 13.2s, min = 12.4s, avg = 12.8s, dev = 0.4s
//tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu           PASSED in 41.5s
  Stats over 2 runs: max = 41.5s, min = 26.8s, avg = 34.1s, dev = 7.4s
//tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu             PASSED in 118.9s
  Stats over 2 runs: max = 118.9s, min = 50.4s, avg = 84.7s, dev = 34.3s
//tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 11.2s
  Stats over 2 runs: max = 11.2s, min = 11.1s, avg = 11.1s, dev = 0.1s
//tensorflow/python/ops:control_flow_ops_test_cpu                        PASSED in 54.0s
  Stats over 2 runs: max = 54.0s, min = 48.2s, avg = 51.1s, dev = 2.9s
//tensorflow/compiler/tests:spacetobatch_op_test_cpu                     PASSED in 11.9s
  Stats over 3 runs: max = 11.9s, min = 11.8s, avg = 11.8s, dev = 0.0s
//tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test    PASSED in 12.5s
  Stats over 3 runs: max = 12.5s, min = 12.2s, avg = 12.3s, dev = 0.2s
//tensorflow/compiler/xla/tests:triangular_solve_test_cpu                PASSED in 53.7s
  Stats over 3 runs: max = 53.7s, min = 50.8s, avg = 52.2s, dev = 1.2s
//tensorflow/core/data/service:thread_safe_buffer_test                   PASSED in 0.1s
  Stats over 3 runs: max = 0.1s, min = 0.1s, avg = 0.1s, dev = 0.0s
//tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 27.6s
  Stats over 3 runs: max = 27.6s, min = 22.6s, avg = 25.9s, dev = 2.3s
//tensorflow/python/data/kernel_tests:unique_test                        PASSED in 25.9s
  Stats over 3 runs: max = 25.9s, min = 23.5s, avg = 24.6s, dev = 1.0s
//tensorflow/python/distribute/coordinator:metric_utils_test             PASSED in 23.3s
  Stats over 3 runs: max = 23.3s, min = 18.6s, avg = 20.6s, dev = 2.0s
//tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu            PASSED in 75.2s
  Stats over 3 runs: max = 75.2s, min = 54.6s, avg = 62.1s, dev = 9.3s
//tensorflow/python/kernel_tests/array_ops:weights_broadcast_test        PASSED in 12.1s
  Stats over 3 runs: max = 12.1s, min = 11.9s, avg = 12.0s, dev = 0.1s
//tensorflow/python/kernel_tests/distributions:util_test_cpu             PASSED in 13.8s
  Stats over 3 runs: max = 13.8s, min = 12.8s, avg = 13.2s, dev = 0.5s
//tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 74.7s
  Stats over 3 runs: max = 74.7s, min = 25.0s, avg = 41.7s, dev = 23.3s
//tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu      PASSED in 21.9s
  Stats over 3 runs: max = 21.9s, min = 17.4s, avg = 19.0s, dev = 2.1s
//tensorflow/compiler/xla/tests:dynamic_ops_test_cpu                     PASSED in 10.7s
  Stats over 4 runs: max = 10.7s, min = 8.9s, avg = 9.7s, dev = 0.8s
//tensorflow/core/kernels:example_parsing_ops_test                       PASSED in 0.6s
  Stats over 4 runs: max = 0.6s, min = 0.5s, avg = 0.6s, dev = 0.0s
//tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 49.9s
  Stats over 4 runs: max = 49.9s, min = 25.7s, avg = 38.8s, dev = 10.0s
//tensorflow/python/data/experimental/kernel_tests:map_and_batch_test    PASSED in 45.3s
  Stats over 4 runs: max = 45.3s, min = 28.1s, avg = 32.8s, dev = 7.2s
//tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 26.3s
  Stats over 4 runs: max = 26.3s, min = 14.6s, avg = 20.4s, dev = 5.2s
//tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test  PASSED in 25.1s
  Stats over 4 runs: max = 25.1s, min = 12.5s, avg = 17.3s, dev = 4.8s
//tensorflow/python/data/experimental/kernel_tests:sql_dataset_test      PASSED in 54.8s
  Stats over 4 runs: max = 54.8s, min = 43.7s, avg = 48.9s, dev = 4.7s
//tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 56.8s
  Stats over 4 runs: max = 56.8s, min = 54.1s, avg = 55.6s, dev = 1.0s
//tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 55.0s
  Stats over 4 runs: max = 55.0s, min = 23.8s, avg = 37.9s, dev = 13.3s
//tensorflow/python/data/kernel_tests:batch_test                         PASSED in 40.7s
  Stats over 4 runs: max = 40.7s, min = 32.9s, avg = 36.2s, dev = 2.8s
//tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test   PASSED in 18.9s
  Stats over 4 runs: max = 18.9s, min = 12.1s, avg = 15.3s, dev = 3.2s
//tensorflow/python/data/kernel_tests:from_generator_test                PASSED in 33.9s
  Stats over 4 runs: max = 33.9s, min = 17.0s, avg = 25.0s, dev = 6.3s
//tensorflow/python/data/kernel_tests:group_by_window_test               PASSED in 38.7s
  Stats over 4 runs: max = 38.7s, min = 10.3s, avg = 23.4s, dev = 12.3s
//tensorflow/python/data/kernel_tests:ragged_batch_test                  PASSED in 51.4s
  Stats over 4 runs: max = 51.4s, min = 47.9s, avg = 50.1s, dev = 1.4s
//tensorflow/python/data/kernel_tests:skip_test                          PASSED in 29.0s
  Stats over 4 runs: max = 29.0s, min = 19.2s, avg = 23.6s, dev = 4.3s
//tensorflow/python/data/kernel_tests:take_test                          PASSED in 37.6s
  Stats over 4 runs: max = 37.6s, min = 37.0s, avg = 37.4s, dev = 0.2s
//tensorflow/python/data/kernel_tests:take_while_test                    PASSED in 32.4s
  Stats over 4 runs: max = 32.4s, min = 30.6s, avg = 31.6s, dev = 0.6s
//tensorflow/python/data/kernel_tests:text_line_dataset_test             PASSED in 24.0s
  Stats over 4 runs: max = 24.0s, min = 17.7s, avg = 21.0s, dev = 3.0s
//tensorflow/python/data/kernel_tests:zip_test                           PASSED in 16.9s
  Stats over 4 runs: max = 16.9s, min = 14.1s, avg = 15.4s, dev = 1.0s
//tensorflow/python/debug/lib:dumping_callback_test_cpu                  PASSED in 24.2s
  Stats over 4 runs: max = 24.2s, min = 22.0s, avg = 23.1s, dev = 0.8s
//tensorflow/python/distribute:cross_device_ops_test_2gpu                PASSED in 29.3s
  Stats over 4 runs: max = 29.3s, min = 21.3s, avg = 25.0s, dev = 3.0s
//tensorflow/python/distribute:cross_device_ops_test_cpu                 PASSED in 34.7s
  Stats over 4 runs: max = 34.7s, min = 25.0s, avg = 29.7s, dev = 3.9s
//tensorflow/python/framework:convert_to_constants_test                  PASSED in 26.6s
  Stats over 4 runs: max = 26.6s, min = 19.5s, avg = 22.5s, dev = 2.7s
//tensorflow/python/kernel_tests:collective_ops_test_2gpu                PASSED in 39.2s
  Stats over 4 runs: max = 39.2s, min = 37.9s, avg = 38.5s, dev = 0.5s
//tensorflow/python/kernel_tests:collective_ops_test_cpu                 PASSED in 35.5s
  Stats over 4 runs: max = 35.5s, min = 33.3s, avg = 34.2s, dev = 0.8s
//tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu            PASSED in 43.1s
  Stats over 4 runs: max = 43.1s, min = 41.0s, avg = 41.9s, dev = 0.9s
//tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu             PASSED in 85.3s
  Stats over 4 runs: max = 85.3s, min = 31.7s, avg = 54.8s, dev = 22.1s
//tensorflow/python/kernel_tests/array_ops:split_op_test_cpu             PASSED in 30.1s
  Stats over 4 runs: max = 30.1s, min = 11.0s, avg = 18.2s, dev = 7.8s
//tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu               PASSED in 100.5s
  Stats over 4 runs: max = 100.5s, min = 18.9s, avg = 50.3s, dev = 32.7s
//tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 30.6s
  Stats over 4 runs: max = 30.6s, min = 29.5s, avg = 30.2s, dev = 0.4s
//tensorflow/python/kernel_tests/random:random_gamma_test_cpu            PASSED in 111.8s
  Stats over 4 runs: max = 111.8s, min = 10.2s, avg = 54.6s, dev = 44.2s
//tensorflow/python/kernel_tests/signal:window_ops_test_cpu              PASSED in 28.2s
  Stats over 4 runs: max = 28.2s, min = 27.4s, avg = 27.7s, dev = 0.3s
//tensorflow/python/ops:nn_batchnorm_test_cpu                            PASSED in 21.3s
  Stats over 4 runs: max = 21.3s, min = 16.9s, avg = 18.6s, dev = 1.8s
//tensorflow/python/ops:nn_fused_batchnorm_d9m_test_cpu                  PASSED in 18.8s
  Stats over 4 runs: max = 18.8s, min = 18.7s, avg = 18.8s, dev = 0.1s
//tensorflow/python/ops/ragged:ragged_gather_op_test                     PASSED in 104.5s
  Stats over 4 runs: max = 104.5s, min = 44.3s, avg = 71.1s, dev = 21.6s
//tensorflow/python/ops/ragged:ragged_getitem_test                       PASSED in 67.7s
  Stats over 4 runs: max = 67.7s, min = 61.0s, avg = 65.0s, dev = 2.7s
//tensorflow/compiler/tests:async_comp_test_cpu                          PASSED in 11.2s
  Stats over 5 runs: max = 11.2s, min = 8.5s, avg = 9.4s, dev = 1.0s
//tensorflow/compiler/tests:conv3d_test_cpu                              PASSED in 28.2s
  Stats over 5 runs: max = 28.2s, min = 12.4s, avg = 18.9s, dev = 6.1s
//tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test             PASSED in 20.8s
  Stats over 5 runs: max = 20.8s, min = 16.0s, avg = 18.0s, dev = 2.0s
//tensorflow/compiler/tests:depthwise_conv_op_test_cpu                   PASSED in 17.2s
  Stats over 5 runs: max = 17.2s, min = 10.8s, avg = 14.0s, dev = 2.5s
//tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test  PASSED in 19.1s
  Stats over 5 runs: max = 19.1s, min = 13.1s, avg = 15.7s, dev = 2.7s
//tensorflow/compiler/tests:fused_batchnorm_test_cpu                     PASSED in 9.8s
  Stats over 5 runs: max = 9.8s, min = 9.3s, avg = 9.5s, dev = 0.2s
//tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test    PASSED in 10.8s
  Stats over 5 runs: max = 10.8s, min = 10.2s, avg = 10.6s, dev = 0.2s
//tensorflow/compiler/tests:image_ops_jit_compile_test_cpu               PASSED in 11.7s
  Stats over 5 runs: max = 11.7s, min = 9.8s, avg = 10.3s, dev = 0.7s
//tensorflow/compiler/tests:reduce_ops_test_cpu                          PASSED in 12.2s
  Stats over 5 runs: max = 12.2s, min = 11.5s, avg = 11.9s, dev = 0.3s
//tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test         PASSED in 15.2s
  Stats over 5 runs: max = 15.2s, min = 13.1s, avg = 14.0s, dev = 0.7s
//tensorflow/compiler/tests:repeat_op_test_cpu                           PASSED in 33.0s
  Stats over 5 runs: max = 33.0s, min = 21.3s, avg = 29.8s, dev = 4.3s
//tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test          PASSED in 11.1s
  Stats over 5 runs: max = 11.1s, min = 9.6s, avg = 10.1s, dev = 0.5s
//tensorflow/compiler/tests:special_math_test_cpu                        PASSED in 95.8s
  Stats over 5 runs: max = 95.8s, min = 23.1s, avg = 57.4s, dev = 24.7s
//tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test       PASSED in 150.5s
  Stats over 5 runs: max = 150.5s, min = 17.4s, avg = 65.1s, dev = 45.7s
//tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu           PASSED in 34.5s
  Stats over 5 runs: max = 34.5s, min = 18.7s, avg = 28.2s, dev = 6.6s
//tensorflow/core/grappler/optimizers:constant_folding_test              PASSED in 4.1s
  Stats over 5 runs: max = 4.1s, min = 2.6s, avg = 3.3s, dev = 0.7s
//tensorflow/dtensor/python/tests:layout_propagation_test_cpu            PASSED in 12.9s
  Stats over 5 runs: max = 12.9s, min = 11.0s, avg = 12.0s, dev = 0.6s
//tensorflow/dtensor/python/tests:multi_mesh_test_cpu                    PASSED in 22.3s
  Stats over 5 runs: max = 22.3s, min = 10.1s, avg = 19.5s, dev = 4.7s
//tensorflow/python/distribute:mirrored_strategy_test_2gpu               PASSED in 49.2s
  Stats over 5 runs: max = 49.2s, min = 47.0s, avg = 48.3s, dev = 0.8s
//tensorflow/python/distribute:mirrored_strategy_test_cpu                PASSED in 20.5s
  Stats over 5 runs: max = 20.5s, min = 18.6s, avg = 19.7s, dev = 0.7s
//tensorflow/python/distribute:moving_averages_test_2gpu                 PASSED in 18.9s
  Stats over 5 runs: max = 18.9s, min = 16.9s, avg = 17.9s, dev = 0.9s
//tensorflow/python/distribute:moving_averages_test_cpu                  PASSED in 25.2s
  Stats over 5 runs: max = 25.2s, min = 20.2s, avg = 23.2s, dev = 1.9s
//tensorflow/python/distribute:vars_test_2gpu                            PASSED in 19.5s
  Stats over 5 runs: max = 19.5s, min = 17.7s, avg = 18.4s, dev = 0.6s
//tensorflow/python/distribute:vars_test_cpu                             PASSED in 23.0s
  Stats over 5 runs: max = 23.0s, min = 20.6s, avg = 21.9s, dev = 0.8s
//tensorflow/python/eager:device_placement_test_cpu                      PASSED in 12.1s
  Stats over 5 runs: max = 12.1s, min = 9.9s, avg = 11.2s, dev = 0.8s
//tensorflow/python/eager:forwardprop_test_cpu                           PASSED in 178.7s
  Stats over 5 runs: max = 178.7s, min = 63.4s, avg = 101.7s, dev = 40.4s
//tensorflow/python/eager/polymorphic_function:gradients_test_cpu        PASSED in 20.8s
  Stats over 5 runs: max = 20.8s, min = 15.6s, avg = 17.6s, dev = 2.1s
//tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu             PASSED in 59.7s
  Stats over 5 runs: max = 59.7s, min = 38.6s, avg = 49.5s, dev = 7.1s
//tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 33.7s
  Stats over 5 runs: max = 33.7s, min = 31.1s, avg = 32.3s, dev = 0.9s
//tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 55.7s
  Stats over 5 runs: max = 55.7s, min = 51.5s, avg = 53.0s, dev = 1.5s
//tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu    PASSED in 32.0s
  Stats over 5 runs: max = 32.0s, min = 28.9s, avg = 30.9s, dev = 1.2s
//tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 36.3s
  Stats over 5 runs: max = 36.3s, min = 33.6s, avg = 34.9s, dev = 1.1s
//tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 38.8s
  Stats over 5 runs: max = 38.8s, min = 31.7s, avg = 35.9s, dev = 2.4s
//tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 42.4s
  Stats over 5 runs: max = 42.4s, min = 41.2s, avg = 42.0s, dev = 0.4s
//tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 105.0s
  Stats over 5 runs: max = 105.0s, min = 48.6s, avg = 92.9s, dev = 22.1s
//tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 34.6s
  Stats over 5 runs: max = 34.6s, min = 28.7s, avg = 31.7s, dev = 2.3s
//tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 52.3s
  Stats over 5 runs: max = 52.3s, min = 18.1s, avg = 28.1s, dev = 12.4s
//tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 127.3s
  Stats over 5 runs: max = 127.3s, min = 124.2s, avg = 125.7s, dev = 1.2s
//tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu    PASSED in 12.2s
  Stats over 5 runs: max = 12.2s, min = 10.1s, avg = 11.4s, dev = 0.8s
//tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu   PASSED in 39.7s
  Stats over 5 runs: max = 39.7s, min = 39.5s, avg = 39.6s, dev = 0.1s
//tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test      PASSED in 37.4s
  Stats over 5 runs: max = 37.4s, min = 32.2s, avg = 34.0s, dev = 2.0s
//tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test      PASSED in 17.8s
  Stats over 5 runs: max = 17.8s, min = 9.4s, avg = 11.8s, dev = 3.1s
//tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu          PASSED in 34.6s
  Stats over 5 runs: max = 34.6s, min = 10.2s, avg = 16.1s, dev = 9.3s
//tensorflow/python/ops/parallel_for:math_test_cpu                       PASSED in 96.1s
  Stats over 5 runs: max = 96.1s, min = 31.0s, avg = 66.5s, dev = 21.2s
//tensorflow/compiler/tests:scan_ops_test_cpu                            PASSED in 18.0s
  Stats over 6 runs: max = 18.0s, min = 14.7s, avg = 16.9s, dev = 1.1s
//tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test           PASSED in 22.1s
  Stats over 6 runs: max = 22.1s, min = 16.5s, avg = 19.3s, dev = 1.7s
//tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 26.2s
  Stats over 6 runs: max = 26.2s, min = 8.8s, avg = 16.5s, dev = 7.5s
//tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu              PASSED in 75.2s
  Stats over 6 runs: max = 75.2s, min = 12.7s, avg = 28.5s, dev = 21.3s
//tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu         PASSED in 51.0s
  Stats over 6 runs: max = 51.0s, min = 26.2s, avg = 36.6s, dev = 7.8s
//tensorflow/python/ops:accumulate_n_benchmark_cpu                       PASSED in 12.9s
  Stats over 6 runs: max = 12.9s, min = 7.3s, avg = 11.5s, dev = 1.9s
//tensorflow/python/distribute/experimental/rpc:rpc_ops_test             PASSED in 17.3s
  Stats over 7 runs: max = 17.3s, min = 13.1s, avg = 14.7s, dev = 1.4s
//tensorflow/compiler/tests:matrix_diag_ops_test_cpu                     PASSED in 87.3s
  Stats over 8 runs: max = 87.3s, min = 38.2s, avg = 54.2s, dev = 16.6s
//tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test    PASSED in 79.5s
  Stats over 8 runs: max = 79.5s, min = 8.7s, avg = 33.2s, dev = 25.0s
//tensorflow/dtensor/python/tests:input_util_test                        PASSED in 27.9s
  Stats over 8 runs: max = 27.9s, min = 17.7s, avg = 23.5s, dev = 3.4s
//tensorflow/python/data/experimental/kernel_tests:csv_dataset_test      PASSED in 28.8s
  Stats over 8 runs: max = 28.8s, min = 10.4s, avg = 18.3s, dev = 7.2s
//tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 29.4s
  Stats over 8 runs: max = 29.4s, min = 13.2s, avg = 20.6s, dev = 5.4s
//tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 56.2s
  Stats over 8 runs: max = 56.2s, min = 11.0s, avg = 28.8s, dev = 15.9s
//tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 68.3s
  Stats over 8 runs: max = 68.3s, min = 52.6s, avg = 55.5s, dev = 5.1s
//tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 54.0s
  Stats over 8 runs: max = 54.0s, min = 37.9s, avg = 43.8s, dev = 5.7s
//tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 25.2s
  Stats over 8 runs: max = 25.2s, min = 6.6s, avg = 12.4s, dev = 5.9s
//tensorflow/python/data/kernel_tests:filter_test                        PASSED in 26.5s
  Stats over 8 runs: max = 26.5s, min = 18.0s, avg = 21.8s, dev = 2.3s
//tensorflow/python/data/kernel_tests:flat_map_test                      PASSED in 56.9s
  Stats over 8 runs: max = 56.9s, min = 36.7s, avg = 45.4s, dev = 6.7s
//tensorflow/python/data/kernel_tests:shard_test                         PASSED in 24.8s
  Stats over 8 runs: max = 24.8s, min = 20.1s, avg = 22.5s, dev = 1.7s
//tensorflow/python/data/kernel_tests:shuffle_test                       PASSED in 84.3s
  Stats over 8 runs: max = 84.3s, min = 32.5s, avg = 40.8s, dev = 16.7s
//tensorflow/python/data/kernel_tests:tf_record_dataset_test             PASSED in 25.2s
  Stats over 8 runs: max = 25.2s, min = 15.3s, avg = 20.2s, dev = 2.8s
//tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 98.4s
  Stats over 8 runs: max = 98.4s, min = 12.3s, avg = 37.5s, dev = 32.6s
//tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu              PASSED in 56.8s
  Stats over 8 runs: max = 56.8s, min = 34.7s, avg = 48.1s, dev = 7.8s
//tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 81.0s
  Stats over 8 runs: max = 81.0s, min = 62.8s, avg = 73.0s, dev = 6.5s
//tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 139.4s
  Stats over 8 runs: max = 139.4s, min = 36.9s, avg = 77.2s, dev = 37.8s
//tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu   PASSED in 61.2s
  Stats over 8 runs: max = 61.2s, min = 7.4s, avg = 16.3s, dev = 17.6s
//tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu       PASSED in 14.7s
  Stats over 8 runs: max = 14.7s, min = 13.7s, avg = 14.4s, dev = 0.3s
//tensorflow/python/kernel_tests/signal:fft_ops_test_cpu                 PASSED in 47.6s
  Stats over 8 runs: max = 47.6s, min = 30.3s, avg = 39.8s, dev = 6.1s
//tensorflow/python/ops/ragged:dynamic_ragged_shape_test                 PASSED in 57.4s
  Stats over 8 runs: max = 57.4s, min = 36.6s, avg = 44.4s, dev = 6.4s
//tensorflow/python/ops/ragged:ragged_tensor_test                        PASSED in 26.3s
  Stats over 8 runs: max = 26.3s, min = 14.3s, avg = 18.8s, dev = 3.4s
//tensorflow/compiler/tests:bincount_op_test_cpu                         PASSED in 22.6s
  Stats over 10 runs: max = 22.6s, min = 5.5s, avg = 19.7s, dev = 4.8s
//tensorflow/compiler/tests:conv2d_test_cpu                              PASSED in 15.7s
  Stats over 10 runs: max = 15.7s, min = 14.3s, avg = 15.0s, dev = 0.5s
//tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test             PASSED in 10.4s
  Stats over 10 runs: max = 10.4s, min = 9.5s, avg = 10.0s, dev = 0.3s
//tensorflow/compiler/tests:random_ops_test_cpu                          PASSED in 19.1s
  Stats over 10 runs: max = 19.1s, min = 13.1s, avg = 15.8s, dev = 1.8s
//tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test         PASSED in 15.9s
  Stats over 10 runs: max = 15.9s, min = 9.1s, avg = 12.9s, dev = 2.1s
//tensorflow/compiler/tests:stateless_random_ops_test_cpu                PASSED in 86.3s
  Stats over 10 runs: max = 86.3s, min = 42.2s, avg = 67.6s, dev = 18.2s
//tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 87.0s
  Stats over 10 runs: max = 87.0s, min = 44.1s, avg = 66.1s, dev = 17.6s
//tensorflow/compiler/tests:stochastic_cast_op_test_cpu                  PASSED in 32.6s
  Stats over 10 runs: max = 32.6s, min = 26.5s, avg = 28.6s, dev = 1.9s
//tensorflow/compiler/xla/client/lib:svd_test_cpu                        PASSED in 35.9s
  Stats over 10 runs: max = 35.9s, min = 6.5s, avg = 15.2s, dev = 10.7s
//tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu                PASSED in 9.6s
  Stats over 10 runs: max = 9.6s, min = 6.2s, avg = 7.4s, dev = 1.0s
//tensorflow/compiler/xla/service/cpu:cpu_runtime_test                   PASSED in 14.1s
  Stats over 10 runs: max = 14.1s, min = 1.1s, avg = 9.5s, dev = 4.3s
//tensorflow/python/data/kernel_tests:rejection_resample_test            PASSED in 21.7s
  Stats over 10 runs: max = 21.7s, min = 9.7s, avg = 14.3s, dev = 3.9s
//tensorflow/python/distribute:input_lib_type_spec_test_2gpu             PASSED in 23.5s
  Stats over 10 runs: max = 23.5s, min = 11.5s, avg = 17.7s, dev = 4.1s
//tensorflow/python/distribute:input_lib_type_spec_test_cpu              PASSED in 23.1s
  Stats over 10 runs: max = 23.1s, min = 12.4s, avg = 18.0s, dev = 4.0s
//tensorflow/python/framework:config_vgpu_test_2gpu                      PASSED in 14.7s
  Stats over 10 runs: max = 14.7s, min = 13.8s, avg = 14.3s, dev = 0.3s
//tensorflow/python/framework:config_vgpu_test_cpu                       PASSED in 13.1s
  Stats over 10 runs: max = 13.1s, min = 11.1s, avg = 11.8s, dev = 0.7s
//tensorflow/python/framework:function_test_cpu                          PASSED in 56.7s
  Stats over 10 runs: max = 56.7s, min = 8.5s, avg = 15.3s, dev = 14.1s
//tensorflow/python/grappler:cluster_test_cpu                            PASSED in 10.4s
  Stats over 10 runs: max = 10.4s, min = 7.3s, avg = 9.1s, dev = 0.8s
//tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu            PASSED in 16.9s
  Stats over 10 runs: max = 16.9s, min = 11.1s, avg = 14.0s, dev = 1.8s
//tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu          PASSED in 10.9s
  Stats over 10 runs: max = 10.9s, min = 8.8s, avg = 9.8s, dev = 0.8s
//tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 14.2s
  Stats over 10 runs: max = 14.2s, min = 10.1s, avg = 12.3s, dev = 1.2s
//tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 114.2s
  Stats over 10 runs: max = 114.2s, min = 92.3s, avg = 106.2s, dev = 6.1s
//tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu   PASSED in 133.9s
  Stats over 10 runs: max = 133.9s, min = 8.4s, avg = 21.7s, dev = 37.4s
//tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 42.1s
  Stats over 10 runs: max = 42.1s, min = 14.7s, avg = 26.3s, dev = 8.7s
//tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 29.4s
  Stats over 10 runs: max = 29.4s, min = 12.5s, avg = 19.7s, dev = 6.7s
//tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu             PASSED in 28.1s
  Stats over 10 runs: max = 28.1s, min = 10.3s, avg = 15.0s, dev = 6.3s
//tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu                     PASSED in 32.0s
  Stats over 10 runs: max = 32.0s, min = 29.7s, avg = 31.2s, dev = 0.7s
//tensorflow/python/kernel_tests/random:random_index_shuffle_test        PASSED in 14.5s
  Stats over 10 runs: max = 14.5s, min = 13.2s, avg = 13.9s, dev = 0.4s
//tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu    PASSED in 111.4s
  Stats over 10 runs: max = 111.4s, min = 30.7s, avg = 70.4s, dev = 38.3s
//tensorflow/python/ops:special_math_ops_test_cpu                        PASSED in 56.9s
  Stats over 10 runs: max = 56.9s, min = 10.9s, avg = 18.5s, dev = 13.1s
//tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu            PASSED in 14.4s
  Stats over 10 runs: max = 14.4s, min = 9.7s, avg = 11.6s, dev = 1.4s
//tensorflow/python/ops/numpy_ops/tests:np_indexing_test                 PASSED in 118.6s
  Stats over 10 runs: max = 118.6s, min = 109.8s, avg = 113.2s, dev = 2.8s
//tensorflow/python/ops/ragged:ragged_tensor_supported_values_test       PASSED in 22.9s
  Stats over 10 runs: max = 22.9s, min = 20.4s, avg = 21.4s, dev = 0.8s
//tensorflow/python/saved_model:load_test_cpu                            PASSED in 57.4s
  Stats over 10 runs: max = 57.4s, min = 33.0s, avg = 38.6s, dev = 6.8s
//tensorflow/python/distribute/failure_handling:failure_handler_test      FLAKY, failed in 3 out of 11 in 114.7s
  Stats over 11 runs: max = 114.7s, min = 45.1s, avg = 86.1s, dev = 25.8s
  /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_4_of_8/test_attempts/attempt_1.log
  /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_5_of_8/test_attempts/attempt_1.log
  /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_1_of_8/test_attempts/attempt_1.log
//tensorflow/compiler/tests:fft_test_cpu                                 PASSED in 27.7s
  Stats over 12 runs: max = 27.7s, min = 14.8s, avg = 20.5s, dev = 5.1s
//tensorflow/compiler/xla/service:triangular_solve_expander_test         PASSED in 5.6s
  Stats over 12 runs: max = 5.6s, min = 2.3s, avg = 3.4s, dev = 0.9s
//tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 22.0s
  Stats over 12 runs: max = 22.0s, min = 8.6s, avg = 13.0s, dev = 4.5s
//tensorflow/python/data/kernel_tests:choose_from_datasets_test          PASSED in 15.1s
  Stats over 12 runs: max = 15.1s, min = 9.9s, avg = 11.9s, dev = 1.5s
//tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu            PASSED in 15.3s
  Stats over 12 runs: max = 15.3s, min = 4.7s, avg = 10.7s, dev = 3.1s
//tensorflow/python/distribute:multi_process_runner_test_2gpu            PASSED in 226.2s
  Stats over 12 runs: max = 226.2s, min = 16.8s, avg = 53.0s, dev = 58.0s
//tensorflow/python/distribute:multi_process_runner_test_cpu             PASSED in 231.5s
  Stats over 12 runs: max = 231.5s, min = 18.6s, avg = 58.9s, dev = 59.6s
//tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 21.1s
  Stats over 15 runs: max = 21.1s, min = 12.5s, avg = 18.1s, dev = 2.0s
//tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu                PASSED in 103.1s
  Stats over 15 runs: max = 103.1s, min = 54.1s, avg = 61.7s, dev = 12.2s
//tensorflow/compiler/tests:ftrl_test_cpu                                PASSED in 12.4s
  Stats over 16 runs: max = 12.4s, min = 6.5s, avg = 8.8s, dev = 1.9s
//tensorflow/compiler/tests:ternary_ops_test_cpu                         PASSED in 14.0s
  Stats over 16 runs: max = 14.0s, min = 4.6s, avg = 9.3s, dev = 2.9s
//tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test        PASSED in 36.5s
  Stats over 16 runs: max = 36.5s, min = 7.9s, avg = 16.7s, dev = 9.4s
//tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 28.2s
  Stats over 16 runs: max = 28.2s, min = 22.5s, avg = 25.3s, dev = 1.9s
//tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 27.2s
  Stats over 16 runs: max = 27.2s, min = 4.9s, avg = 14.5s, dev = 5.3s
//tensorflow/python/data/kernel_tests:snapshot_test                      PASSED in 43.0s
  Stats over 16 runs: max = 43.0s, min = 14.7s, avg = 27.3s, dev = 10.0s
//tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 38.8s
  Stats over 16 runs: max = 38.8s, min = 10.9s, avg = 18.6s, dev = 6.4s
//tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test       PASSED in 15.3s
  Stats over 16 runs: max = 15.3s, min = 9.4s, avg = 10.7s, dev = 1.4s
//tensorflow/python/kernel_tests/signal:dct_ops_test_cpu                 PASSED in 15.8s
  Stats over 16 runs: max = 15.8s, min = 8.3s, avg = 12.3s, dev = 2.6s
//tensorflow/python/ops:image_ops_test_cpu                               PASSED in 22.7s
  Stats over 16 runs: max = 22.7s, min = 11.6s, avg = 15.9s, dev = 3.1s
//tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 79.2s
  Stats over 17 runs: max = 79.2s, min = 7.9s, avg = 26.7s, dev = 28.1s
//tensorflow/python/data/kernel_tests:map_test                           PASSED in 39.5s
  Stats over 19 runs: max = 39.5s, min = 11.4s, avg = 23.8s, dev = 6.9s
//tensorflow/compiler/tests:pooling_ops_3d_test_cpu                      PASSED in 10.3s
  Stats over 20 runs: max = 10.3s, min = 6.5s, avg = 7.8s, dev = 1.2s
//tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test     PASSED in 10.7s
  Stats over 20 runs: max = 10.7s, min = 7.1s, avg = 8.6s, dev = 1.1s
//tensorflow/compiler/tests:pooling_ops_test_cpu                         PASSED in 13.9s
  Stats over 20 runs: max = 13.9s, min = 7.7s, avg = 8.9s, dev = 1.4s
//tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test        PASSED in 15.2s
  Stats over 20 runs: max = 15.2s, min = 7.4s, avg = 9.1s, dev = 1.8s
//tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu   PASSED in 9.0s
  Stats over 20 runs: max = 9.0s, min = 7.7s, avg = 8.2s, dev = 0.4s
//tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 16.0s
  Stats over 20 runs: max = 16.0s, min = 13.0s, avg = 14.7s, dev = 0.8s
//tensorflow/compiler/xla/tests:dot_operation_test_cpu                   PASSED in 11.4s
  Stats over 20 runs: max = 11.4s, min = 9.6s, avg = 10.3s, dev = 0.5s
//tensorflow/compiler/xla/tests:prng_test_cpu                            PASSED in 8.3s
  Stats over 20 runs: max = 8.3s, min = 7.1s, avg = 7.8s, dev = 0.3s
//tensorflow/compiler/xla/tests:reduce_window_test_cpu                   PASSED in 40.9s
  Stats over 20 runs: max = 40.9s, min = 6.2s, avg = 15.7s, dev = 10.7s
//tensorflow/python/autograph/tests:loop_control_flow_test               PASSED in 106.0s
  Stats over 20 runs: max = 106.0s, min = 99.8s, avg = 103.0s, dev = 2.0s
//tensorflow/python/kernel_tests:metrics_test                            PASSED in 40.5s
  Stats over 20 runs: max = 40.5s, min = 8.4s, avg = 20.7s, dev = 9.5s
//tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu  PASSED in 10.2s
  Stats over 20 runs: max = 10.2s, min = 7.0s, avg = 8.5s, dev = 1.1s
//tensorflow/python/kernel_tests/data_structures:barrier_ops_test        PASSED in 15.0s
  Stats over 20 runs: max = 15.0s, min = 6.3s, avg = 9.1s, dev = 2.5s
//tensorflow/python/kernel_tests/linalg:eig_op_test                      PASSED in 57.4s
  Stats over 20 runs: max = 57.4s, min = 9.0s, avg = 22.2s, dev = 15.7s
//tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu             PASSED in 137.9s
  Stats over 20 runs: max = 137.9s, min = 40.7s, avg = 77.7s, dev = 24.2s
//tensorflow/python/kernel_tests/linalg:norm_op_test_cpu                 PASSED in 13.1s
  Stats over 20 runs: max = 13.1s, min = 5.2s, avg = 9.1s, dev = 2.7s
//tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu            PASSED in 17.0s
  Stats over 20 runs: max = 17.0s, min = 8.0s, avg = 12.3s, dev = 2.3s
//tensorflow/python/kernel_tests/linalg:qr_op_test_cpu                   PASSED in 177.4s
  Stats over 20 runs: max = 177.4s, min = 45.4s, avg = 106.8s, dev = 41.1s
//tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu     PASSED in 44.0s
  Stats over 20 runs: max = 44.0s, min = 14.7s, avg = 26.6s, dev = 8.7s
//tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu       PASSED in 28.4s
  Stats over 20 runs: max = 28.4s, min = 9.8s, avg = 18.7s, dev = 7.2s
//tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu             PASSED in 20.7s
  Stats over 20 runs: max = 20.7s, min = 17.1s, avg = 19.0s, dev = 1.1s
//tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu          PASSED in 72.3s
  Stats over 20 runs: max = 72.3s, min = 9.3s, avg = 33.1s, dev = 21.8s
//tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu           PASSED in 24.4s
  Stats over 20 runs: max = 24.4s, min = 14.2s, avg = 16.7s, dev = 2.1s
//tensorflow/python/data/experimental/kernel_tests/service:local_workers_test PASSED in 24.0s
  Stats over 24 runs: max = 24.0s, min = 8.4s, avg = 17.8s, dev = 4.1s
//tensorflow/python/data/kernel_tests:interleave_test                    PASSED in 27.6s
  Stats over 24 runs: max = 27.6s, min = 9.8s, avg = 16.9s, dev = 4.0s
//tensorflow/python/data/kernel_tests:sample_from_datasets_test          PASSED in 20.8s
  Stats over 24 runs: max = 20.8s, min = 6.5s, avg = 11.9s, dev = 4.2s
//tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu           PASSED in 9.0s
  Stats over 25 runs: max = 9.0s, min = 7.3s, avg = 8.0s, dev = 0.5s
//tensorflow/compiler/xla/tests:select_and_scatter_test_cpu              PASSED in 35.8s
  Stats over 25 runs: max = 35.8s, min = 7.0s, avg = 12.0s, dev = 7.3s
//tensorflow/compiler/xla/tests:convolution_variants_test_cpu            PASSED in 9.6s
  Stats over 30 runs: max = 9.6s, min = 6.6s, avg = 8.1s, dev = 0.7s
//tensorflow/compiler/xla/tests:iota_test_cpu                            PASSED in 17.4s
  Stats over 30 runs: max = 17.4s, min = 12.2s, avg = 14.6s, dev = 1.2s
//tensorflow/compiler/xla/tests:params_test_cpu                          PASSED in 9.3s
  Stats over 30 runs: max = 9.3s, min = 7.2s, avg = 7.9s, dev = 0.4s
//tensorflow/compiler/xla/tests:reshape_test_cpu                         PASSED in 11.4s
  Stats over 30 runs: max = 11.4s, min = 7.6s, avg = 9.1s, dev = 0.9s
//tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu             PASSED in 40.4s
  Stats over 30 runs: max = 40.4s, min = 4.6s, avg = 17.8s, dev = 11.8s
//tensorflow/compiler/xla/tests:reduce_test_cpu                          PASSED in 11.8s
  Stats over 31 runs: max = 11.8s, min = 9.0s, avg = 10.7s, dev = 0.7s
//tensorflow/compiler/xla/tests:scalar_computations_test_cpu             PASSED in 34.4s
  Stats over 32 runs: max = 34.4s, min = 7.4s, avg = 16.2s, dev = 10.3s
//tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 28.9s
  Stats over 32 runs: max = 28.9s, min = 10.7s, avg = 18.0s, dev = 4.4s
//tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 71.8s
  Stats over 32 runs: max = 71.8s, min = 32.9s, avg = 48.1s, dev = 14.2s
//tensorflow/compiler/xla/tests:batch_normalization_test_cpu             PASSED in 11.3s
  Stats over 40 runs: max = 11.3s, min = 8.5s, avg = 10.2s, dev = 0.8s
//tensorflow/compiler/xla/tests:bfloat16_test_cpu                        PASSED in 11.1s
  Stats over 40 runs: max = 11.1s, min = 9.8s, avg = 10.3s, dev = 0.4s
//tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu  PASSED in 10.2s
  Stats over 40 runs: max = 10.2s, min = 7.2s, avg = 8.8s, dev = 0.7s
//tensorflow/compiler/xla/tests:slice_test_cpu                           PASSED in 63.9s
  Stats over 40 runs: max = 63.9s, min = 6.7s, avg = 29.7s, dev = 26.2s
//tensorflow/core/kernels:stochastic_cast_op_test                        PASSED in 11.1s
  Stats over 48 runs: max = 11.1s, min = 0.4s, avg = 0.8s, dev = 1.5s
//tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 77.7s
  Stats over 50 runs: max = 77.7s, min = 19.7s, avg = 37.6s, dev = 18.5s
//tensorflow/compiler/tests:sort_ops_test_cpu                            PASSED in 42.6s
  Stats over 50 runs: max = 42.6s, min = 3.5s, avg = 12.5s, dev = 8.2s
//tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test           PASSED in 48.3s
  Stats over 50 runs: max = 48.3s, min = 3.5s, avg = 12.4s, dev = 9.2s
//tensorflow/compiler/tests:unary_ops_test_cpu                           PASSED in 17.9s
  Stats over 50 runs: max = 17.9s, min = 3.6s, avg = 7.2s, dev = 3.2s
//tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test          PASSED in 27.0s
  Stats over 50 runs: max = 27.0s, min = 3.6s, avg = 7.9s, dev = 4.6s
//tensorflow/compiler/xla/tests:conv_depthwise_test_cpu                  PASSED in 11.4s
  Stats over 50 runs: max = 11.4s, min = 8.5s, avg = 9.6s, dev = 0.7s
//tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu       PASSED in 13.2s
  Stats over 50 runs: max = 13.2s, min = 9.7s, avg = 11.5s, dev = 1.0s
//tensorflow/compiler/xla/tests:convolution_test_cpu                     PASSED in 16.4s
  Stats over 50 runs: max = 16.4s, min = 9.3s, avg = 12.4s, dev = 1.8s
//tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 16.2s
  Stats over 50 runs: max = 16.2s, min = 5.2s, avg = 10.5s, dev = 3.2s
//tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 11.6s
  Stats over 50 runs: max = 11.6s, min = 3.9s, avg = 6.8s, dev = 2.3s
//tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 10.9s
  Stats over 50 runs: max = 10.9s, min = 3.9s, avg = 7.3s, dev = 2.2s
//tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu      PASSED in 33.8s
  Stats over 50 runs: max = 33.8s, min = 8.9s, avg = 22.8s, dev = 6.9s
//tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu             PASSED in 17.8s
  Stats over 50 runs: max = 17.8s, min = 4.2s, avg = 7.6s, dev = 2.5s
//tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu       PASSED in 17.5s
  Stats over 50 runs: max = 17.5s, min = 4.0s, avg = 6.9s, dev = 3.6s

Executed 3894 out of 3894 tests: 3894 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.