==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 5 of 8): Running tests under Python 3.11.6: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I0111 23:03:15.915710 281473085568032 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0111 23:03:15.916200 281473085568032 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0111 23:03:15.916521 281473085568032 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0111 23:03:15.916744 281473085568032 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee01a58a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0111 23:03:16.160208 281473085568032 polymorphic_function.py:157] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee01a58a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee01a58a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0111 23:03:16.195313 281473085568032 polymorphic_function.py:157] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee01a58a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0111 23:03:16.364234 281473085568032 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I0111 23:03:17.091547 281473085568032 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I0111 23:03:17.346139 281473085568032 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I0111 23:03:17.746250 281473085568032 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I0111 23:03:17.941878 281473085568032 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I0111 23:03:18.249095 281473085568032 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I0111 23:03:18.461717 281473085568032 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I0111 23:03:18.666017 281473085568032 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I0111 23:03:18.666632 281473085568032 failure_handler_test.py:245] Training finished. INFO:tensorflow:sending sigterm I0111 23:03:18.801325 281470145851872 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I0111 23:03:18.801833 281473085568032 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 3.0s I0111 23:03:18.802414 281473085568032 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 3.0s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24116 I0111 23:03:18.818952 281473085568032 test_util.py:3999] Using local port 24116 INFO:tensorflow:Using local port 24115 I0111 23:03:18.820784 281473085568032 test_util.py:3999] Using local port 24115 INFO:tensorflow:Using local port 24114 I0111 23:03:18.824246 281473085568032 test_util.py:3999] Using local port 24114 INFO:tensorflow:Using local port 24113 I0111 23:03:18.826193 281473085568032 test_util.py:3999] Using local port 24113 INFO:tensorflow:Cluster starting. I0111 23:03:23.516938 281473085568032 failure_handler_test.py:297] Cluster starting. [worker-0]: I0111 23:03:23.781628 281472925856800 multi_process_runner.py:840] Subprocess with PID 1721483 (worker, 0) is now being started. [worker-0]: I0111 23:03:23.782088 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0111 23:03:23.787090 281472925856800 multi_process_runner.py:840] Subprocess with PID 1721537 (worker, 1) is now being started. [worker-3]: I0111 23:03:23.789858 281472925856800 multi_process_runner.py:840] Subprocess with PID 1721617 (worker, 3) is now being started. [worker-1]: I0111 23:03:23.787557 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0111 23:03:23.790317 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0111 23:03:23.792954 281472925856800 multi_process_runner.py:840] Subprocess with PID 1721609 (worker, 2) is now being started. [worker-2]: I0111 23:03:23.793404 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-2]: 2024-01-11 23:03:23.848032: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24114 [worker-0]: 2024-01-11 23:03:23.880874: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24116 [worker-3]: 2024-01-11 23:03:23.903000: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24113 [worker-0]: 2024-01-11 23:03:23.911327: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 9499101314529963751 [worker-0]: 2024-01-11 23:03:23.911380: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2024-01-11 23:03:23.911391: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:1 [worker-0]: /job:worker/replica:0/task:0 [worker-0]: /job:worker/replica:0/task:2 [worker-1]: 2024-01-11 23:03:23.921097: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24115 [worker-0]: 2024-01-11 23:03:23.948016: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 467877906186738427 [worker-0]: 2024-01-11 23:03:23.948069: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-0]: 2024-01-11 23:03:23.948081: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:0 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: /job:worker/replica:0/task:2 [worker-1]: I0000 00:00:1705014203.949233 1721537 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:23.949146: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 6752783689959994394 [worker-0]: 2024-01-11 23:03:23.949167: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-0]: 2024-01-11 23:03:23.949177: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:2 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1705014203.949336 1721483 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: I0000 00:00:1705014203.924100 1721617 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:24.856096: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 12578771561968829130 [worker-0]: 2024-01-11 23:03:24.856136: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-2]: I0000 00:00:1705014204.857114 1721609 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0111 23:03:24.867762 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0111 23:03:24.860569 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0111 23:03:24.865955 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0111 23:03:24.877171 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0111 23:03:24.927341 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0111 23:03:24.928733 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0111 23:03:24.929530 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0111 23:03:24.925480 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0111 23:03:24.926060 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: I0111 23:03:24.926301 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:24.937707 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0111 23:03:24.940291 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:24.940557 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0111 23:03:24.950576 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0111 23:03:24.951339 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:24.951572 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0111 23:03:25.105317 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0111 23:03:25.109918 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0111 23:03:25.110236 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0111 23:03:25.110538 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0111 23:03:25.110724 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0111 23:03:25.116084 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0111 23:03:25.117427 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0111 23:03:25.117685 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0111 23:03:25.117981 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0111 23:03:25.118168 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0111 23:03:25.125485 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0111 23:03:25.134665 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0111 23:03:25.136516 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:25.136785 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0111 23:03:25.137089 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0111 23:03:25.137275 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0111 23:03:25.261211 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0111 23:03:25.261554 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0111 23:03:25.261862 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0111 23:03:25.262048 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:25.285888 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:25.372821 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:25.454777 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:25.467608 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:25.579541 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:25.609573 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:25.607656 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:25.621802 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:25.778192 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:25.779994 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:25.800643 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:25.817409 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:26.211042 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:26.231332 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:26.251209 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:26.271389 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:26.520408 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:26.530228 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:26.532000 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:26.575852 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff741477e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:26.637733 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff741477e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a65d00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:26.638592 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a65d00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a644a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:26.638884 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a644a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a68360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:26.644756 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a68360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:26.648173 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:26.648797 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:26.652313 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:26.654760 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a65760> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a64220> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:26.754549 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a64220> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:26.746785 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a65760> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a64180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:26.762658 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a64180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:26.774870 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:26.791951 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a680e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:26.866417 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a680e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:26.912099 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.022773 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:27.149736 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.141539 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:27.171705 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:27.173138 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:27.404194 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.426081 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:27.449586 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:27.464953 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:27.529710 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.535204 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:27.536632 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:27.530085 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:27.615706 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.615825 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:27.621917 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:27.630328 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:27.725782 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:27.733065 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.738436 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:27.745161 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:27.833173 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:27.834056 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.845620 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:27.850577 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:27.945938 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:27.956715 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:27.970240 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:27.980139 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.124994 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.140161 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.151318 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.205057 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0111 23:03:28.309417 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0111 23:03:28.327978 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-0]: I0111 23:03:28.325982 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0111 23:03:28.335408 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.337948 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.343556 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.369679 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.416155 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.526555 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.545598 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.560810 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.550261 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.663982 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.685173 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.676636 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.719501 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.883243 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.899363 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.910373 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.899331 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.029338 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.029301 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.059538 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.060831 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.170116 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.171283 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.175930 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.189991 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.410290 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.410710 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.425402 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.430656 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.596147 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.605223 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.629891 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.740023 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.864582 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.869608 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.865714 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.895617 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.028153 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.031902 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.039363 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.046220 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.121135 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.122482 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.165627 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.189258 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.279390 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.300129 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.290237 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.300033 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.394711 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.400128 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.420151 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.430320 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.517729 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.529531 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.540333 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.560301 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.662464 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.665023 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.664910 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.656723 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0111 23:03:30.797044 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0111 23:03:30.799633 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0111 23:03:30.811288 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.818704 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0111 23:03:30.819112 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.816040 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.834725 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.826159 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.937683 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.959808 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.959988 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.980395 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.131410 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.119404 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.159532 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.156175 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.222625 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.226293 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.233032 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.228183 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.320913 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.343915 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.342743 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.330745 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.423323 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.434497 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.440907 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.442354 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.555417 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.567396 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.572645 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.579553 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.642999 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.643000 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.645510 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.643103 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.755249 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.759943 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.763029 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.755296 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I0111 23:03:31.793634 281473085568032 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I0111 23:03:31.793986 281473085568032 failure_handler_test.py:306] sigterm sent [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-2]: I0111 23:03:31.805869 281472925856800 failure_handling.py:710] Member 2 has received termination notice. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.860810 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.864018 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I0111 23:03:31.875435 281472925856800 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 40 [worker-2]: I0111 23:03:31.877415 281472925856800 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 40 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.873691 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-1]: I0111 23:03:31.880368 281447185183200 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-2]: I0111 23:03:31.885249 281472925856800 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I0111 23:03:31.886193 281472925856800 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-0]: I0111 23:03:31.888767 281447419802080 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I0111 23:03:31.895885 281446455439840 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I0111 23:03:31.896095 281472925856800 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I0111 23:03:31.900794 281472925856800 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-3]: I0111 23:03:31.915188 281446858093024 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.965815 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0111 23:03:32.051361 281472925856800 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0111 23:03:32.051839 281472925856800 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0111 23:03:32.051732 281472925856800 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0111 23:03:32.051510 281472925856800 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/workertemp_1/fh_ckpt [worker-1]: I0111 23:03:32.486454 281472925856800 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/workertemp_1/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I0111 23:03:32.487829 281472925856800 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0111 23:03:32.488018 281472925856800 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/workertemp_3/fh_ckpt [worker-3]: I0111 23:03:32.489802 281472925856800 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/workertemp_3/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/fh_ckpt [worker-3]: I0111 23:03:32.491733 281472925856800 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/workertemp_2/fh_ckpt [worker-0]: I0111 23:03:32.491764 281472925856800 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/fh_ckpt [worker-2]: I0111 23:03:32.491768 281472925856800 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2erj9oylt/tmp7vpnjksa/workertemp_2/fh_ckpt [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0111 23:03:32.491948 281472925856800 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0111 23:03:32.493721 281472925856800 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0111 23:03:32.493947 281472925856800 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0111 23:03:32.494020 281472925856800 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0111 23:03:32.494237 281472925856800 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0111 23:03:33.794662 281473085568032 failure_handler_test.py:309] restarting workers [worker-0]: I0111 23:03:34.498649 281472925856800 multi_process_runner.py:840] Subprocess with PID 1739291 (worker, 0) is now being started. INFO:tensorflow:workers restarted [worker-0]: I0111 23:03:34.499139 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' I0111 23:03:34.502819 281473085568032 failure_handler_test.py:313] workers restarted [worker-2]: I0111 23:03:34.521444 281472925856800 multi_process_runner.py:840] Subprocess with PID 1739513 (worker, 2) is now being started. [worker-0]: 2024-01-11 23:03:34.536553: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24116 [worker-2]: I0111 23:03:34.521938 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0111 23:03:34.550331 281472925856800 multi_process_runner.py:840] Subprocess with PID 1739377 (worker, 1) is now being started. [worker-0]: 2024-01-11 23:03:34.554692: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 5128946101574256561 [worker-0]: 2024-01-11 23:03:34.554749: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-1]: I0111 23:03:34.550796 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2024-01-11 23:03:34.554761: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:1 [worker-0]: /job:worker/replica:0/task:2 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1705014214.555008 1739291 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: I0111 23:03:34.559189 281472925856800 multi_process_runner.py:840] Subprocess with PID 1739596 (worker, 3) is now being started. [worker-3]: I0111 23:03:34.559680 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24116", "localhost:24115", "localhost:24114", "localhost:24113"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2024-01-11 23:03:34.572615: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24114 [worker-0]: 2024-01-11 23:03:34.585243: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 1219006053460222129 [worker-0]: 2024-01-11 23:03:34.585301: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-0]: 2024-01-11 23:03:34.585313: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:1 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-2]: I0000 00:00:1705014214.585688 1739513 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2024-01-11 23:03:34.606226: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24115 [worker-0]: 2024-01-11 23:03:34.631582: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 7210749442263808534 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2024-01-11 23:03:34.631638: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-1]: I0000 00:00:1705014214.631836 1739377 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:34.631650: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-3]: 2024-01-11 23:03:34.655991: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24113 [worker-0]: 2024-01-11 23:03:34.663747: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 1254614907204079216 [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: I0000 00:00:1705014214.663963 1739596 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:34.663798: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0111 23:03:34.666131 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0111 23:03:34.666135 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0111 23:03:34.666213 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0111 23:03:34.666419 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0111 23:03:34.723076 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: I0111 23:03:34.723040 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: I0111 23:03:34.720656 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0111 23:03:34.721235 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0111 23:03:34.721475 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0111 23:03:34.723563 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0111 23:03:34.723799 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0111 23:03:34.723602 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:34.723842 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0111 23:03:34.749184 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0111 23:03:34.749814 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:34.750057 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24116', 'localhost:24115', 'localhost:24114', 'localhost:24113']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0111 23:03:34.833577 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0111 23:03:34.847719 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0111 23:03:34.865353 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0111 23:03:34.848479 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0111 23:03:34.865705 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: I0111 23:03:34.848720 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: W0111 23:03:34.866012 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-2]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0111 23:03:34.849034 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-2]: INFO:tensorflow:Start training at 40 [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0111 23:03:34.866197 281472925856800 failure_handler_test.py:197] Start training at 40 [worker-1]: INFO:tensorflow:Start training at 40 [worker-2]: INFO:tensorflow:training restarted [worker-1]: I0111 23:03:34.849222 281472925856800 failure_handler_test.py:197] Start training at 40 [worker-2]: I0111 23:03:34.886475 281472925856800 failure_handler_test.py:207] training restarted [worker-1]: INFO:tensorflow:training restarted [worker-1]: I0111 23:03:34.851531 281472925856800 failure_handler_test.py:207] training restarted [worker-3]: I0111 23:03:34.847645 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0111 23:03:34.848369 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0111 23:03:34.848625 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0111 23:03:34.849017 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 40 [worker-3]: I0111 23:03:34.849243 281472925856800 failure_handler_test.py:197] Start training at 40 [worker-3]: INFO:tensorflow:training restarted [worker-3]: I0111 23:03:34.851441 281472925856800 failure_handler_test.py:207] training restarted [worker-0]: I0111 23:03:34.833356 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0111 23:03:34.834152 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:34.834391 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0111 23:03:34.834687 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 40 [worker-0]: I0111 23:03:34.834869 281472925856800 failure_handler_test.py:197] Start training at 40 [worker-0]: INFO:tensorflow:training restarted [worker-0]: I0111 23:03:34.837133 281472925856800 failure_handler_test.py:207] training restarted [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.044648 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.062919 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.077558 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.104846 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.198018 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.198044 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.210919 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.217727 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.498688 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.518430 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.512554 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.542668 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.650141 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.656137 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.660878 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.684056 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.779491 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.772572 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.790150 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.783768 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a64c20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:35.942913 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a64c20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a3bf60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a69580> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:35.943472 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a3bf60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:35.955294 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a69580> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0111 23:03:35.943318 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-1]: I0111 23:03:35.955750 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-2]: I0111 23:03:35.943925 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a69260> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:35.980350 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a69260> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0111 23:03:35.980743 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.979571 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.999470 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.010057 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.009704 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a696c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a642c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:36.161159 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a642c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:36.155075 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a696c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.182917 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a69120> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:36.168489 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a69120> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a69120> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:36.185396 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a69120> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.193415 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.209444 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.195347 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.503314 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.570214 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.586848 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.626153 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.711744 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.722876 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.740513 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.729645 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.876847 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.884101 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.879873 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.889772 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.059096 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.064672 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.069871 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.083044 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.154439 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.154604 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.165780 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.190083 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.274782 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.279917 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.277184 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.275830 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.349148 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.348317 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.348404 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.366090 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.494913 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.508559 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.529969 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.563848 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.646984 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.651401 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.653283 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.682330 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.765498 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.775646 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.796704 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.790203 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.894899 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.895184 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.896526 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.909589 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.009571 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.025666 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.004632 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.030010 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.183462 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.189602 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.205951 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.199423 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0111 23:03:38.289237 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0111 23:03:38.291391 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0111 23:03:38.291866 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.300331 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.305941 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.350917 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0111 23:03:38.366333 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.399615 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.557787 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.609706 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.631669 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.622816 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.740514 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.768117 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.759862 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.772576 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.900447 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.890259 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.893909 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.899615 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.109863 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.114855 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.129481 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.139928 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.309623 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.319949 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.309802 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.339978 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.420822 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.434699 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.446542 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.449993 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.650009 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.669970 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.680027 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.689545 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.777053 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.784114 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.799671 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.845923 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.931414 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.932752 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.936802 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.942840 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.020337 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.049907 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.049191 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.069617 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.205427 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.240817 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.249865 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.259357 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.321445 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.342365 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.342419 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.342398 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.429616 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.431838 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.419644 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.449647 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.728232 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.731771 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.750388 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.751236 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0111 23:03:40.826042 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0111 23:03:40.835839 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-0]: I0111 23:03:40.833770 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.845975 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.845318 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.836357 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.869944 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.883207 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.989668 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.993702 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.989921 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.016639 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.111767 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.111266 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.130028 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.146147 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.219414 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.241079 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.239669 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.261607 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.363445 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.373322 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.380272 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.449635 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.575934 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.600692 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.620384 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.595933 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.758003 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.795956 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.789425 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.816127 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.970013 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.989867 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.969599 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.980085 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.091266 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.109574 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.120739 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.129906 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.269766 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.276469 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.299964 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.299664 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.523935 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.518302 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.529139 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.531234 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.691790 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.700865 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.703265 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.742351 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.869904 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.869534 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.879898 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.900140 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.102943 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.126265 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.149437 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.146027 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.270332 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.281674 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.281417 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.306329 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-1]: I0111 23:03:43.361849 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I0111 23:03:43.363577 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.373184 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.374308 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-3]: I0111 23:03:43.375397 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-0]: I0111 23:03:43.385368 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.401090 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.406173 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.497118 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.519396 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.526222 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.539966 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.624973 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.654640 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.656157 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.653832 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.753168 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.757806 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.770023 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.761862 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.854961 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.865389 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.871457 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.867179 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.933703 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.933949 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.934579 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.944236 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.039726 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.043237 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.058063 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.055173 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.165013 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.167400 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.171640 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.167823 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.280384 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.284331 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.288387 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.299613 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.433746 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.437428 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.448878 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.470743 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.706485 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.761785 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.739963 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.772018 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.906573 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.912121 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.915001 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.913429 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.019917 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.037262 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.042054 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.081865 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.166653 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.164760 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.182544 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.203444 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.353816 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.370949 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.370138 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.389852 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-0]: I0111 23:03:45.509630 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-3]: I0111 23:03:45.507987 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-2]: I0111 23:03:45.520329 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I0111 23:03:45.536108 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.526010 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.539844 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.550182 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.561320 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.721117 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.711740 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.726727 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.740702 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.810847 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.824422 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.850049 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.876825 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.963447 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.969972 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.958348 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.999683 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.104468 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.120599 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.136536 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.189873 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.479913 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.510509 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.508017 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.501288 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.663948 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.679358 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.719842 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.725970 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.872601 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.873416 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.899437 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.906567 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.055690 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.071004 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.071544 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.103698 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.204727 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.229336 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.238162 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.253304 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.510254 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.515129 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.534688 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.550660 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.665687 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.671642 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.677453 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.729533 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.801121 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.805829 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.819578 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.820240 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.060081 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.072064 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.089830 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.085906 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.270117 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.259565 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.289833 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.290154 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-0]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I0111 23:03:48.346471 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-0]: I0111 23:03:48.345131 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-3]: I0111 23:03:48.343526 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0111 23:03:48.344531 281472925856800 failure_handler_test.py:245] Training finished. [worker-2]: INFO:tensorflow:epoch 7 finished [worker-2]: I0111 23:03:48.349253 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0111 23:03:48.349935 281472925856800 failure_handler_test.py:245] Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0111 23:03:48.350618 281472925856800 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0111 23:03:48.356198 281472925856800 failure_handler_test.py:245] Training finished. I0111 23:03:49.535207 281473085568032 multi_process_runner.py:646] worker-0 exit code: 0 I0111 23:03:49.535517 281473085568032 multi_process_runner.py:646] worker-1 exit code: 0 I0111 23:03:49.535671 281473085568032 multi_process_runner.py:646] worker-2 exit code: 0 I0111 23:03:49.535810 281473085568032 multi_process_runner.py:646] worker-3 exit code: 0 I0111 23:03:49.538054 281473085568032 multi_process_runner.py:662] Joining log reading threads. I0111 23:03:49.538316 281473085568032 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 31.03s I0111 23:03:49.829052 281473085568032 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 31.03s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24006 I0111 23:03:49.851890 281473085568032 test_util.py:3999] Using local port 24006 INFO:tensorflow:Using local port 24002 I0111 23:03:49.853865 281473085568032 test_util.py:3999] Using local port 24002 INFO:tensorflow:Using local port 24001 I0111 23:03:49.855640 281473085568032 test_util.py:3999] Using local port 24001 INFO:tensorflow:Using local port 24000 I0111 23:03:49.857266 281473085568032 test_util.py:3999] Using local port 24000 INFO:tensorflow:Cluster starting. I0111 23:03:50.155937 281473085568032 failure_handler_test.py:297] Cluster starting. [worker-0]: I0111 23:03:50.277192 281472925856800 multi_process_runner.py:840] Subprocess with PID 1766947 (worker, 0) is now being started. [worker-0]: I0111 23:03:50.277659 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24006", "localhost:24002", "localhost:24001", "localhost:24000"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0111 23:03:50.507371 281472925856800 multi_process_runner.py:840] Subprocess with PID 1767106 (worker, 1) is now being started. [worker-1]: I0111 23:03:50.507831 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24006", "localhost:24002", "localhost:24001", "localhost:24000"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-1]: 2024-01-11 23:03:50.645123: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24002 [worker-0]: 2024-01-11 23:03:50.685180: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24006 [worker-2]: I0111 23:03:50.687774 281472925856800 multi_process_runner.py:840] Subprocess with PID 1767111 (worker, 2) is now being started. [worker-3]: I0111 23:03:50.696611 281472925856800 multi_process_runner.py:840] Subprocess with PID 1767399 (worker, 3) is now being started. [worker-2]: I0111 23:03:50.688225 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24006", "localhost:24002", "localhost:24001", "localhost:24000"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0111 23:03:50.697077 281472925856800 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24006", "localhost:24002", "localhost:24001", "localhost:24000"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2024-01-11 23:03:50.728992: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 10894073681103505259 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-1]: I0000 00:00:1705014230.730246 1767106 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:50.729046: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2024-01-11 23:03:50.729059: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:0 [worker-0]: /job:worker/replica:0/task:2 [worker-0]: 2024-01-11 23:03:50.729255: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 6565658827954026818 [worker-0]: 2024-01-11 23:03:50.729269: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-0]: 2024-01-11 23:03:50.729278: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:2 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1705014230.730563 1766947 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2024-01-11 23:03:50.762024: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24000 [worker-0]: 2024-01-11 23:03:50.775094: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 2869376721634833341 [worker-0]: 2024-01-11 23:03:50.775145: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-0]: 2024-01-11 23:03:50.775157: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:2 [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: I0000 00:00:1705014230.776673 1767399 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2024-01-11 23:03:50.882233: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24001 [worker-0]: 2024-01-11 23:03:50.895179: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 3962404374221636370 [worker-0]: 2024-01-11 23:03:50.895229: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-2]: I0000 00:00:1705014230.895648 1767111 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0111 23:03:50.955960 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0111 23:03:50.956656 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0111 23:03:50.966322 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0111 23:03:50.954790 281472925856800 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0111 23:03:51.197067 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0111 23:03:51.200050 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:51.200833 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0111 23:03:51.222335 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0111 23:03:51.223753 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0111 23:03:51.224528 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0111 23:03:51.245721 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0111 23:03:51.247155 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:51.247938 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0111 23:03:51.320366 281472925856800 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0111 23:03:51.321679 281472925856800 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0111 23:03:51.322455 281472925856800 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24006', 'localhost:24002', 'localhost:24001', 'localhost:24000']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0111 23:03:51.464461 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0111 23:03:51.464657 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0111 23:03:51.466303 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0111 23:03:51.465682 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0111 23:03:51.465938 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0111 23:03:51.466224 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0111 23:03:51.485530 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-1]: I0111 23:03:51.485781 281472925856800 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0111 23:03:51.466407 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-3]: I0111 23:03:51.485874 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0111 23:03:51.486180 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0111 23:03:51.486371 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0111 23:03:51.505733 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0111 23:03:51.506079 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0111 23:03:51.506389 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0111 23:03:51.506577 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0111 23:03:51.518539 281472925856800 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:51.518894 281472925856800 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0111 23:03:51.519212 281472925856800 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0111 23:03:51.519401 281472925856800 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:51.763138 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:51.754069 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:51.838736 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:51.907521 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:52.152384 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:52.179625 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:52.200323 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:52.210626 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:52.348422 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:52.370304 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:52.369927 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:52.409613 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:52.992019 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:53.010880 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:53.030823 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:53.050021 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:53.370471 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:53.570664 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:53.630900 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:53.670474 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a6b6a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:53.871943 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a6b6a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a6b6a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a6b4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:53.895292 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a6b4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:53.877667 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a6b6a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:53.899445 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed6a69bc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:53.905865 281472925856800 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffed6a69bc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:53.904567 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:53.930681 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:53.940660 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a68ae0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a6a2a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a6b7e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:54.163619 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a6b7e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:54.158339 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a6a2a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:54.152488 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a68ae0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed6a6b920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:54.169334 281472925856800 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffed6a6b920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:54.189702 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:54.226958 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:54.212276 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:54.429840 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:54.551034 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:54.557835 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:54.569986 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:54.582948 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:54.703473 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:54.719370 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:54.729340 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:54.752297 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:54.929467 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:54.929662 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:54.956626 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:54.997256 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:55.159421 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:55.170475 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:55.209187 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:55.209188 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:55.389413 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:55.389339 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:55.409154 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:55.415789 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:55.529697 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:55.529531 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:55.539194 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:55.609098 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:55.699517 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:55.709255 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:55.729247 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:55.739261 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:55.886514 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:55.899662 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:55.899436 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:55.917176 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0111 23:03:56.074521 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0111 23:03:56.075902 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-3]: I0111 23:03:56.073916 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:56.083146 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0111 23:03:56.086147 281472925856800 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:56.095755 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:56.120410 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:56.149440 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:56.261287 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:56.274675 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:56.303746 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:56.301382 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:56.765532 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:56.794507 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:56.810032 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:56.819859 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:56.904723 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:56.895169 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:56.899394 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:56.919230 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:56.986138 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:56.986151 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:56.991562 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.019429 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.102749 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.102597 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.102542 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.112837 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.184555 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.188878 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.195270 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.195253 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.281101 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.282076 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.281360 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.299385 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.371326 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.371316 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.371309 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.371421 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.432005 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.431988 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.432617 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.448165 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.506172 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.512004 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.513885 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.516200 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.573096 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.573584 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.589506 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.589529 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.661145 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.661378 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.661741 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.663749 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.727011 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.735245 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.735448 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.726843 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.793722 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.793950 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.795239 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.795427 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: I0111 23:03:57.844828 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-3]: I0111 23:03:57.844635 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-2]: I0111 23:03:57.845077 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-1]: I0111 23:03:57.844968 281472925856800 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.853967 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.854115 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.854161 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.854502 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.911933 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.911962 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.912178 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.929564 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.012046 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.012308 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.013785 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.013831 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.072347 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.072348 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.072729 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.097402 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.179144 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.188977 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.203062 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.289387 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0111 23:03:58.415998 281473085568032 failure_handler_test.py:302] sending sigterm [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.358585 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.358614 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.358536 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker [worker-0]: I0111 23:03:58.413219 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.413322 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.359075 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.413113 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 25.48s I0111 23:04:15.312880 281473085568032 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 25.48s [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ====================================================================== [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='checkpoint', strategy_option='MWMS_multi_worker') [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ---------------------------------------------------------------------- [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process [worker-0]: I0111 23:03:58.474247 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ---------------------------------------------------------------------- Ran 3 tests in 59.515s [worker-2]: I0111 23:03:58.413382 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.474231 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 FAILED (errors=1) [worker-1]: I0111 23:03:58.475754 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.531483 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.474509 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.531535 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.530187 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.639050 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.530406 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.643676 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.639155 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.699376 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.630563 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.701256 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.699325 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.757114 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.758900 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.699756 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.756997 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.815504 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.813717 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.757304 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.873480 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.876128 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.814049 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.931922 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.933754 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-3]: I0111 23:03:58.980626 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-1]: I0111 23:03:58.981006 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-2]: I0111 23:03:58.873818 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.988570 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.932305 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.989760 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.980997 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.045846 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.044705 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.988885 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.813771 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.100097 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.101345 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.045075 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.873496 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.157294 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.156324 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.101003 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.932014 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.211316 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.212307 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-2]: I0111 23:03:59.156664 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.980787 281472925856800 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.270654 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.211624 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.272464 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.988608 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.332437 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.270879 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.334314 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.044746 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.332702 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.389660 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.100151 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.446223 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.156401 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.389915 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.502299 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.211396 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.446415 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.562128 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.270790 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.502591 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.390724 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.620615 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.677028 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.447184 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.332463 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.738306 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.389676 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.503855 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.798597 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.562460 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.446222 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.620941 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.564795 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.852563 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.502383 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.622228 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.677417 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.680154 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.738619 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.740882 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.860125 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.562238 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.798885 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.801376 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.852928 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-3]: I0111 23:03:59.921550 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: I0111 23:03:59.620826 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.853094 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-0]: I0111 23:03:59.677197 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.860475 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.863297 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.981342 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.738358 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.921908 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.044507 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.798718 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.923747 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-2]: I0111 23:03:59.981717 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.106559 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.852739 281472925856800 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.984306 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.044844 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.167231 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.860175 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.048034 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.106905 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.226712 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.921595 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.109133 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.167597 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.286822 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.981444 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.227060 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.170222 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.343737 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.044664 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.229408 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.287217 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.106616 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.402820 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.288715 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.167426 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.344122 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.459572 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.345417 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.515557 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.404762 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.460927 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.573181 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.517011 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.629555 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.402712 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.459918 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.515897 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.573532 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.630334 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.687337 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0111 23:04:00.740697 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-0]: I0111 23:04:00.226773 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.748267 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.286978 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.808573 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.343855 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.869497 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.402478 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.574852 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.926975 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.687048 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-2]: I0111 23:04:00.984215 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.631132 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.459650 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.740353 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.040644 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.689330 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.515615 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.748035 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.096387 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: I0111 23:04:00.573325 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.808391 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.740850 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.153597 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.869158 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.629669 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.750720 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.926581 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.687115 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.811203 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.210851 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.871287 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.983778 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.740508 281472925856800 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.040068 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.266564 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.095988 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.323074 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.153338 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.379374 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.210504 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.439578 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.498605 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.558089 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I0111 23:04:01.611437 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.266128 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.618978 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.323155 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.678329 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.378980 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.738941 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.439150 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.498224 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.796090 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.557666 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.856735 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.611071 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-2]: I0111 23:04:01.915415 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.618641 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.975108 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.677957 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.035463 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.738589 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.095155 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.795688 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.855923 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.155837 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.915008 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.214611 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.974717 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.273352 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.035510 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.332377 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.094794 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.155447 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.391407 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.214238 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.450700 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.928413 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.985727 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.041377 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.272913 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.097678 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.154890 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.211921 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.267609 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.323997 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.381571 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.441539 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.331981 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.500486 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.390964 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.560203 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-3]: I0111 23:04:02.450371 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.611594 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.502158 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-1]: I0111 23:04:01.621459 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.509777 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.680954 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.569328 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.740684 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.626566 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.798197 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.685090 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.857878 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.917522 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.977245 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.744618 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.037565 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.801830 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.860653 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.097906 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.748050 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.918307 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.157832 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: I0111 23:04:00.808309 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.975399 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.502531 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.216603 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.869287 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.510143 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.031238 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.275128 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.569999 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.926743 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.088094 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.334181 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.143635 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.393176 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.199202 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.626932 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.255521 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.452605 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.685490 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-3]: I0111 23:04:03.310663 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.502668 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-3]: INFO:tensorflow:epoch 7 finished [worker-2]: I0111 23:04:02.744558 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.362954 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.512279 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.363687 281472925856800 failure_handler_test.py:245] Training finished. [worker-2]: I0111 23:04:02.802240 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.571931 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.860959 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.628530 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.918625 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.983943 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.040245 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.096119 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.153338 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.211070 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.266328 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.687468 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.975821 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.322878 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.746978 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.031600 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.379110 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.803094 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.088502 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.439319 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.862392 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.144085 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.498333 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.920397 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.199597 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.557845 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.976703 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-2]: I0111 23:04:03.255526 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.611233 281472925856800 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.033416 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.311030 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.619173 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: I0111 23:04:03.089067 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.363347 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.678120 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-1]: I0111 23:04:03.145097 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.364089 281472925856800 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.738751 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.200383 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.795812 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.255872 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.856039 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.312916 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.363472 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-0]: I0111 23:04:01.915193 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.364506 281472925856800 failure_handler_test.py:245] Training finished. [worker-0]: I0111 23:04:01.974841 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.035239 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.094895 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.155653 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.214394 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.273097 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.332270 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.391130 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.450473 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-0]: I0111 23:04:02.502324 281472925856800 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.509874 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.569334 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.626721 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.685250 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.744763 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.802026 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.860765 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.918392 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.975577 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.031434 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.088257 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.143795 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.199340 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.255679 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.310820 281472925856800 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I0111 23:04:03.363127 281472925856800 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0111 23:04:03.363783 281472925856800 failure_handler_test.py:245] Training finished. ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 1 of 8): Running tests under Python 3.11.6: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_error_propagation INFO:tensorflow:Using local port 24131 I0111 23:03:15.817130 281473280668704 test_util.py:3999] Using local port 24131 INFO:tensorflow:Using local port 24130 I0111 23:03:15.825825 281473280668704 test_util.py:3999] Using local port 24130 INFO:tensorflow:Using local port 24129 I0111 23:03:15.828418 281473280668704 test_util.py:3999] Using local port 24129 INFO:tensorflow:Using local port 24128 I0111 23:03:15.830513 281473280668704 test_util.py:3999] Using local port 24128 INFO:tensorflow:Cluster starting. I0111 23:03:22.302809 281473280668704 failure_handler_test.py:387] Cluster starting. [worker-0]: I0111 23:03:22.406972 281473065645088 multi_process_runner.py:840] Subprocess with PID 1716511 (worker, 0) is now being started. [worker-0]: I0111 23:03:22.407420 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24131", "localhost:24130", "localhost:24129", "localhost:24128"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2024-01-11 23:03:22.486258: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24131 [worker-2]: I0111 23:03:22.501669 281473065645088 multi_process_runner.py:840] Subprocess with PID 1716626 (worker, 2) is now being started. [worker-3]: I0111 23:03:22.507800 281473065645088 multi_process_runner.py:840] Subprocess with PID 1716952 (worker, 3) is now being started. [worker-2]: I0111 23:03:22.502196 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24131", "localhost:24130", "localhost:24129", "localhost:24128"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0111 23:03:22.508375 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24131", "localhost:24130", "localhost:24129", "localhost:24128"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0111 23:03:22.581567 281473065645088 multi_process_runner.py:840] Subprocess with PID 1716603 (worker, 1) is now being started. [worker-1]: I0111 23:03:22.582087 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24131", "localhost:24130", "localhost:24129", "localhost:24128"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2024-01-11 23:03:22.591281: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 13928950115153802066 [worker-0]: 2024-01-11 23:03:22.591337: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2024-01-11 23:03:22.591349: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:1 [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:2 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1705014202.595885 1716511 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2024-01-11 23:03:22.667113: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24130 [worker-0]: 2024-01-11 23:03:22.675840: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 12372798779150442819 [worker-0]: 2024-01-11 23:03:22.675890: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-0]: 2024-01-11 23:03:22.675902: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:2 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-1]: I0000 00:00:1705014202.676575 1716603 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2024-01-11 23:03:22.726083: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24129 [worker-0]: 2024-01-11 23:03:22.742024: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 7866019917334180088 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2024-01-11 23:03:22.742078: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-2]: I0000 00:00:1705014202.742265 1716626 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:22.742089: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-3]: 2024-01-11 23:03:22.756306: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24128 [worker-0]: 2024-01-11 23:03:22.825638: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 855382639529797141 [worker-0]: 2024-01-11 23:03:22.825681: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: I0000 00:00:1705014202.825964 1716952 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0111 23:03:22.833525 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0111 23:03:22.830642 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0111 23:03:22.846900 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0111 23:03:22.868920 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0111 23:03:22.897541 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0111 23:03:22.898138 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:22.898384 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0111 23:03:22.953432 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0111 23:03:22.954047 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0111 23:03:22.954290 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0111 23:03:22.959218 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0111 23:03:22.959776 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0111 23:03:22.960014 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0111 23:03:22.961975 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0111 23:03:22.962664 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:22.962914 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24131', 'localhost:24130', 'localhost:24129', 'localhost:24128']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0111 23:03:23.077929 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0111 23:03:23.083987 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:23.084525 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0111 23:03:23.084311 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0111 23:03:23.085289 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-2]: Instructions for updating: [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: I0111 23:03:23.085527 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: W0111 23:03:23.084670 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-0]: W0111 23:03:23.085815 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I0111 23:03:23.084861 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0111 23:03:23.085997 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0111 23:03:23.117149 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0111 23:03:23.106195 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0111 23:03:23.107858 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0111 23:03:23.108124 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0111 23:03:23.108452 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0111 23:03:23.108639 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0111 23:03:23.135443 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0111 23:03:23.135864 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0111 23:03:23.136267 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0111 23:03:23.136463 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Error reported to Coordinator: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: Traceback (most recent call last): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-2]: yield [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-2]: [worker-0]: 2024-01-11 23:03:23.251349: E external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:1018] /job:worker/replica:0/task:2 has been set to ERROR in coordination service: RESOURCE_EXHAUSTED: in user code: [worker-0]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: ResourceExhaustedError: Running out of resources [worker-1]: E0000 00:00:1705014203.252654 1718568 coordination_service_agent.cc:749] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-1]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: I0111 23:03:23.245917 281447092711904 coordinator.py:213] Error reported to Coordinator: in user code: [worker-0]: ResourceExhaustedError: Running out of resources [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: E0000 00:00:1705014203.251957 1716831 coordination_service_agent.cc:749] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-3]: E0000 00:00:1705014203.255281 1718622 coordination_service_agent.cc:749] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-2]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: ResourceExhaustedError: Running out of resources [worker-2]: ResourceExhaustedError: Running out of resources [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: Traceback (most recent call last): [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: 2024-01-11 23:03:23.252715: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-2]: yield [worker-1]: [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-0]: 2024-01-11 23:03:23.252009: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-3]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-3]: 2024-01-11 23:03:23.255352: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: I0111 23:03:23.250405 281473065645088 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: E0000 00:00:1705014203.250672 1716626 coordination_service_agent.cc:749] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: 2024-01-11 23:03:23.250783: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: in user code: [worker-2]: [worker-3]: I0111 23:03:23.303384 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: I0000 00:00:1705014203.250827 1716626 coordination_service_agent.cc:438] Reporting error to coordination service: RESOURCE_EXHAUSTED: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:23.312011 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:23.336309 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: 2024-01-11 23:03:23.375163: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: 2024-01-11 23:03:23.385109: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-0]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: [worker-0]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 135, in _main [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-3]: [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-3]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-0]: [worker-3]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-0]: [worker-3]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-3]: [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: File "", line 1, in [worker-0]: [Op:__inference_train_step_40] [worker-3]: [worker-0]: I0111 23:03:23.390613 281473065645088 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main [worker-0]: [worker-3]: [worker-0]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 135, in _main [worker-0]: File "", line 1, in [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 135, in _main [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: [worker-0]: [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-3]: [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-3]: [Op:__inference_train_step_38] [worker-0]: [worker-3]: I0111 23:03:23.380865 281473065645088 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-0]: [worker-3]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-3]: [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: File "", line 1, in [worker-0]: [Op:__inference_train_step_40] [worker-3]: [worker-0]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main [worker-0]: I0111 23:03:23.391025 281473065645088 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 135, in _main [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: [Op:__inference_train_step_38] [worker-3]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-1]: Exception in thread PeerTerminationWatcher-1: [worker-3]: I0111 23:03:23.381352 281473065645088 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-1]: Traceback (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 982, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1229, in _watch_step_to_save_key [worker-1]: step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY) [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 819, in get_config_key_value [worker-1]: pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key, [worker-1]: tensorflow.python.framework.errors_impl.UnavailableError: Socket closed [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/GetKeyValue: [worker-1]: :{"created":"@1705014203.424153361","description":"Error received from peer ipv6:[::1]:24131","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14} [worker-1]: 2024-01-11 23:03:23.586273: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-1]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: [worker-1]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-1]: File "", line 1, in [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 135, in _main [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-1]: [worker-1]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: [Op:__inference_train_step_38] [worker-1]: I0111 23:03:23.591783 281473065645088 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: [worker-1]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-1]: File "", line 1, in [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 274, in main [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/spawn.py", line 135, in _main [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-1]: [worker-1]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: [Op:__inference_train_step_38] [worker-1]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-1]: I0111 23:03:23.592218 281473065645088 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. I0111 23:03:24.396098 281473280668704 multi_process_runner.py:646] worker-0 exit code: 0 I0111 23:03:24.396446 281473280668704 multi_process_runner.py:646] worker-1 exit code: 0 I0111 23:03:24.396627 281473280668704 multi_process_runner.py:646] worker-2 exit code: 0 I0111 23:03:24.396807 281473280668704 multi_process_runner.py:646] worker-3 exit code: 0 I0111 23:03:24.400436 281473280668704 multi_process_runner.py:662] Joining log reading threads. I0111 23:03:24.400768 281473280668704 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_error_propagation): 8.7s I0111 23:03:24.508929 281473280668704 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_error_propagation): 8.7s [ OK ] PreemptionCheckpointTest.test_error_propagation [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I0111 23:03:24.630347 281473280668704 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0111 23:03:24.630743 281473280668704 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0111 23:03:24.631064 281473280668704 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0111 23:03:24.631259 281473280668704 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffeeb9e8360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0111 23:03:24.834560 281473280668704 polymorphic_function.py:157] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffeeb9e8360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffeeb9e8360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0111 23:03:24.848946 281473280668704 polymorphic_function.py:157] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffeeb9e8360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0111 23:03:24.975775 281473280668704 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I0111 23:03:25.191481 281473280668704 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I0111 23:03:25.396545 281473280668704 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I0111 23:03:25.596711 281473280668704 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I0111 23:03:25.832468 281473280668704 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I0111 23:03:26.126227 281473280668704 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:sending sigterm I0111 23:03:26.519415 281470330597856 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I0111 23:03:26.523116 281473280668704 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:Termination caught in main thread on preempted worker I0111 23:03:26.524937 281473280668704 failure_handling.py:1159] Termination caught in main thread on preempted worker INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. I0111 23:03:26.538563 281473280668704 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmpj0b533yc/fh_ckpt I0111 23:03:26.582512 281473280668704 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmpj0b533yc/fh_ckpt INFO:tensorflow:Continue training for the grace period. I0111 23:03:26.582818 281473280668704 failure_handling.py:1134] Continue training for the grace period. INFO:tensorflow:epoch 6 finished I0111 23:03:26.634070 281473280668704 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I0111 23:03:27.045319 281473280668704 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I0111 23:03:27.045911 281473280668704 failure_handler_test.py:245] Training finished. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 2.54s I0111 23:03:27.047003 281473280668704 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 2.54s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24057 I0111 23:03:27.051808 281473280668704 test_util.py:3999] Using local port 24057 INFO:tensorflow:Using local port 24056 I0111 23:03:27.085123 281473280668704 test_util.py:3999] Using local port 24056 INFO:tensorflow:Using local port 24055 I0111 23:03:27.087740 281473280668704 test_util.py:3999] Using local port 24055 INFO:tensorflow:Using local port 24054 I0111 23:03:27.089356 281473280668704 test_util.py:3999] Using local port 24054 INFO:tensorflow:Cluster starting. I0111 23:03:27.277502 281473280668704 failure_handler_test.py:297] Cluster starting. [worker-0]: I0111 23:03:27.377272 281473065645088 multi_process_runner.py:840] Subprocess with PID 1730183 (worker, 0) is now being started. [worker-0]: I0111 23:03:27.377731 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0111 23:03:27.508559 281473065645088 multi_process_runner.py:840] Subprocess with PID 1730197 (worker, 1) is now being started. [worker-2]: I0111 23:03:27.509482 281473065645088 multi_process_runner.py:840] Subprocess with PID 1730271 (worker, 2) is now being started. [worker-1]: I0111 23:03:27.508988 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0111 23:03:27.509882 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0111 23:03:27.541838 281473065645088 multi_process_runner.py:840] Subprocess with PID 1730284 (worker, 3) is now being started. [worker-3]: I0111 23:03:27.542321 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2024-01-11 23:03:27.583390: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24057 [worker-1]: 2024-01-11 23:03:27.588467: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24056 [worker-2]: 2024-01-11 23:03:27.592559: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24055 [worker-0]: 2024-01-11 23:03:27.635331: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 4802260119000176123 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2024-01-11 23:03:27.635392: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-2]: I0000 00:00:1705014207.635679 1730271 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:27.635403: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:0 [worker-0]: /job:worker/replica:0/task:2 [worker-0]: /job:worker/replica:0/task:3 [worker-0]: 2024-01-11 23:03:27.635440: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 6636777478390718590 [worker-0]: 2024-01-11 23:03:27.635463: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-0]: 2024-01-11 23:03:27.635473: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:0 [worker-0]: /job:worker/replica:0/task:3 [worker-0]: 2024-01-11 23:03:27.638355: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 13674155809882070483 [worker-0]: 2024-01-11 23:03:27.638382: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-0]: 2024-01-11 23:03:27.638392: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1705014207.638595 1730183 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: I0000 00:00:1705014207.638990 1730197 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2024-01-11 23:03:27.665487: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24054 [worker-0]: 2024-01-11 23:03:27.695134: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 17831682876844563464 [worker-0]: 2024-01-11 23:03:27.695187: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: I0000 00:00:1705014207.705000 1730284 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0111 23:03:27.716851 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0111 23:03:27.717011 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0111 23:03:27.718269 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0111 23:03:27.737415 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-2]: I0111 23:03:27.768275 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: I0111 23:03:27.773184 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0111 23:03:27.773709 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0111 23:03:27.773946 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0111 23:03:27.768776 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:27.769010 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0111 23:03:27.860545 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0111 23:03:27.862707 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0111 23:03:27.863493 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0111 23:03:27.879956 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0111 23:03:27.880518 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:27.880752 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0111 23:03:28.044034 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0111 23:03:28.063191 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0111 23:03:28.076409 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:28.076721 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0111 23:03:28.077012 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0111 23:03:28.077196 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0111 23:03:28.072788 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0111 23:03:28.081539 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0111 23:03:28.095661 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0111 23:03:28.095980 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0111 23:03:28.096270 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0111 23:03:28.096457 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0111 23:03:28.141044 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0111 23:03:28.141359 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0111 23:03:28.141646 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0111 23:03:28.141832 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0111 23:03:28.195439 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0111 23:03:28.195772 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0111 23:03:28.196068 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0111 23:03:28.196256 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.507286 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.530178 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.584637 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.582628 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.676654 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.682265 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.693649 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.699545 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:28.834896 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:28.849474 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:28.865877 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:28.847008 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.249647 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.259479 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.259470 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.275971 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.429564 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.429523 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.429549 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.456660 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefb8b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:29.566566 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefb8b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.575283 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefb0540> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefb9da0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:29.591868 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefb9da0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:29.585870 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefb0540> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefb4b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:29.594054 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefb4b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.601948 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.637037 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.672488 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefba480> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb44a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:29.781598 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb44a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:29.776229 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefba480> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb2e80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:29.787243 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb2e80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefba840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.809244 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.799273 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0111 23:03:29.792858 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefba840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.846633 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.832879 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:29.919049 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:29.919083 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:29.923562 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:29.953463 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.022330 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.033919 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.076186 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.069153 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.154745 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.169155 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.171327 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.180315 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.280442 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.293519 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.309280 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.300379 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.432190 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.429281 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.445983 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.449840 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.554078 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.552678 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.589718 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.605814 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.708714 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.710047 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.749262 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.729199 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.834536 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.839320 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.860378 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.870007 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: I0111 23:03:30.927747 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-2]: I0111 23:03:30.927875 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0111 23:03:30.929057 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0111 23:03:30.926617 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:30.943642 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:30.948393 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:30.950243 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:30.959163 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.090510 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.090593 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.100308 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.115866 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.217497 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.226385 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.246068 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.265136 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.393886 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.394190 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.383077 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.395223 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I0111 23:03:31.416825 281473280668704 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I0111 23:03:31.417191 281473280668704 failure_handler_test.py:306] sigterm sent [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-2]: I0111 23:03:31.418114 281473065645088 failure_handling.py:710] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I0111 23:03:31.465515 281473065645088 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 20 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:31.473788 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:31.475437 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.470653 281473065645088 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 20 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I0111 23:03:31.476507 281447428256224 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-2]: I0111 23:03:31.478548 281473065645088 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:31.487611 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I0111 23:03:31.495482 281473065645088 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-0]: I0111 23:03:31.496231 281447294038496 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-1]: I0111 23:03:31.501185 281449156178400 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-3]: I0111 23:03:31.491864 281447495365088 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I0111 23:03:31.515558 281473065645088 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I0111 23:03:31.516728 281473065645088 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:31.525558 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0111 23:03:31.599853 281473065645088 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0111 23:03:31.600979 281473065645088 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0111 23:03:31.615551 281473065645088 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0111 23:03:31.606155 281473065645088 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/workertemp_3/fh_ckpt [worker-3]: I0111 23:03:31.646341 281473065645088 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/workertemp_3/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0111 23:03:31.648320 281473065645088 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0111 23:03:31.648536 281473065645088 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/fh_ckpt [worker-0]: I0111 23:03:31.652062 281473065645088 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/fh_ckpt [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0111 23:03:31.653849 281473065645088 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0111 23:03:31.654057 281473065645088 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/workertemp_2/fh_ckpt [worker-2]: I0111 23:03:31.661413 281473065645088 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/workertemp_2/fh_ckpt [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: I0111 23:03:31.662793 281473065645088 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0111 23:03:31.663003 281473065645088 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/workertemp_1/fh_ckpt [worker-1]: I0111 23:03:31.663779 281473065645088 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b903hrqxs3u/tmp5y13auev/workertemp_1/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I0111 23:03:31.665657 281473065645088 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0111 23:03:31.665890 281473065645088 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0111 23:03:33.417845 281473280668704 failure_handler_test.py:309] restarting workers INFO:tensorflow:workers restarted I0111 23:03:34.507538 281473280668704 failure_handler_test.py:313] workers restarted [worker-0]: I0111 23:03:34.506700 281473065645088 multi_process_runner.py:840] Subprocess with PID 1739290 (worker, 0) is now being started. [worker-1]: I0111 23:03:34.515249 281473065645088 multi_process_runner.py:840] Subprocess with PID 1739388 (worker, 1) is now being started. [worker-2]: I0111 23:03:34.520107 281473065645088 multi_process_runner.py:840] Subprocess with PID 1739511 (worker, 2) is now being started. [worker-2]: I0111 23:03:34.520627 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: I0111 23:03:34.507238 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0111 23:03:34.515792 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0111 23:03:34.543333 281473065645088 multi_process_runner.py:840] Subprocess with PID 1739618 (worker, 3) is now being started. [worker-0]: 2024-01-11 23:03:34.544636: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24057 [worker-3]: I0111 23:03:34.543801 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24057", "localhost:24056", "localhost:24055", "localhost:24054"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: 2024-01-11 23:03:34.564610: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24056 [worker-0]: 2024-01-11 23:03:34.567016: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 13285365465095418661 [worker-2]: 2024-01-11 23:03:34.576690: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24055 [worker-0]: 2024-01-11 23:03:34.567069: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2024-01-11 23:03:34.567081: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:2 [worker-0]: /job:worker/replica:0/task:1 [worker-0]: /job:worker/replica:0/task:3 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1705014214.567254 1739290 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2024-01-11 23:03:34.587664: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24054 [worker-0]: 2024-01-11 23:03:34.592867: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 17833813922043554971 [worker-0]: 2024-01-11 23:03:34.592912: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-0]: 2024-01-11 23:03:34.592923: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:1 [worker-0]: /job:worker/replica:0/task:3 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-2]: I0000 00:00:1705014214.593114 1739511 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:34.635612: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 14204857543762961586 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2024-01-11 23:03:34.635665: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-1]: I0000 00:00:1705014214.636062 1739388 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: I0000 00:00:1705014214.636069 1739618 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:34.635677: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: 2024-01-11 23:03:34.635714: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 11519413286760833080 [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0111 23:03:34.638220 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0111 23:03:34.638608 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: 2024-01-11 23:03:34.635735: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0111 23:03:34.646184 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0111 23:03:34.646533 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0111 23:03:34.703294 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0111 23:03:34.703886 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:34.704128 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0111 23:03:34.703327 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0111 23:03:34.703299 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-1]: INFO:tensorflow:Check health not enabled. [worker-0]: I0111 23:03:34.704001 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: I0111 23:03:34.703958 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0111 23:03:34.704248 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0111 23:03:34.704207 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:34.695705 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0111 23:03:34.696343 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:34.696590 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24057', 'localhost:24056', 'localhost:24055', 'localhost:24054']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0111 23:03:34.801437 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0111 23:03:34.801394 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I0111 23:03:34.802310 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0111 23:03:34.802248 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-0]: I0111 23:03:34.802596 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: Instructions for updating: [worker-2]: I0111 23:03:34.802505 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: W0111 23:03:34.802928 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start training at 20 [worker-1]: I0111 23:03:34.811372 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I0111 23:03:34.803126 281473065645088 failure_handler_test.py:197] Start training at 20 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0111 23:03:34.812515 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:training restarted [worker-2]: W0111 23:03:34.802814 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0111 23:03:34.812929 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:34.814223 281473065645088 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: I0111 23:03:34.815813 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0111 23:03:34.878177 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0111 23:03:34.878548 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0111 23:03:34.878883 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 20 [worker-3]: I0111 23:03:34.879086 281473065645088 failure_handler_test.py:197] Start training at 20 [worker-3]: INFO:tensorflow:training restarted [worker-3]: I0111 23:03:34.906466 281473065645088 failure_handler_test.py:207] training restarted [worker-2]: INFO:tensorflow:Start training at 20 [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I0111 23:03:34.803001 281473065645088 failure_handler_test.py:197] Start training at 20 [worker-1]: Instructions for updating: [worker-2]: INFO:tensorflow:training restarted [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0111 23:03:34.813282 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 20 [worker-1]: I0111 23:03:34.813480 281473065645088 failure_handler_test.py:197] Start training at 20 [worker-1]: INFO:tensorflow:training restarted [worker-1]: I0111 23:03:34.820221 281473065645088 failure_handler_test.py:207] training restarted [worker-2]: I0111 23:03:34.814306 281473065645088 failure_handler_test.py:207] training restarted [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.016608 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.047602 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.072537 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.119726 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.295985 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.277025 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.300260 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.320295 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.614213 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.619486 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.618004 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.645972 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:35.771071 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.774526 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:35.776760 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.778975 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:35.869995 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:35.979708 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.020652 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.009935 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedef89da0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:36.232129 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedef89da0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.241478 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedc4cc400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:36.247303 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedc4cc400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedc4cc2c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:36.255579 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedc4cc2c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefbac00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:36.265347 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefbac00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.271098 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.290396 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.316122 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb8400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:36.425930 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb8400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefbb240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0111 23:03:36.419512 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefbb240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefbbf60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:36.426363 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefbbf60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb80e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:36.445613 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb80e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.469595 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.630926 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.650118 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.655275 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:36.921327 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:36.900394 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:36.930157 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:36.940438 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.186428 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.191082 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.198295 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.234542 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.315980 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.317887 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.318733 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.334517 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0111 23:03:37.409242 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0111 23:03:37.411543 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0111 23:03:37.412698 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0111 23:03:37.414144 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.421413 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.439840 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.454779 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.468474 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.585201 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.590445 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.609928 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.590075 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.671110 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.671096 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.674421 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.688598 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.775679 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.775140 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.789056 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.789830 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:37.935086 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:37.954777 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:37.969767 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:37.955221 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.031919 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.040517 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.050419 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.059795 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.123960 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.126999 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.140090 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.159635 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.241814 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.255003 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.259648 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.300117 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.385385 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.384383 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.420217 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.388115 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.647488 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.653607 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.662172 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.684926 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:38.801190 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:38.805061 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:38.809861 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:38.806259 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.084322 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.114155 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.115777 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.126347 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.330685 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.335772 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.340421 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.345972 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.474732 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.479466 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.480546 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.486051 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.668734 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.700104 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.701892 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.709688 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0111 23:03:39.816075 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0111 23:03:39.825362 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0111 23:03:39.829131 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0111 23:03:39.818024 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.835930 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.845966 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.871006 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.875999 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:39.936479 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:39.939835 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:39.939993 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:39.970224 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.085386 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.089640 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.099896 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.130731 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.205242 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.221943 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.225700 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.256094 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.336234 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.342226 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.356450 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.354108 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.462072 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.469686 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.535979 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.535959 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.600241 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.619944 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.620140 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.730856 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:40.875265 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:40.880596 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:40.920964 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:40.895954 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.075531 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.119549 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.134404 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.140280 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.319921 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.310531 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.339440 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.388656 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.535817 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.544130 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.556896 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.599849 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.734412 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.745944 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.739691 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.756057 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:41.932407 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:41.937348 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:41.932304 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:41.966006 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.055911 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.074647 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.099632 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.120147 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.197985 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.193483 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.209822 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.210007 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0111 23:03:42.266038 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0111 23:03:42.270581 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0111 23:03:42.275455 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0111 23:03:42.315780 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.355954 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.359568 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.369738 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.440126 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.536654 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.568375 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.569618 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.596106 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:42.883592 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:42.894700 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:42.889784 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:42.895955 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.005752 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.005957 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.039817 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.071377 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.163024 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.163621 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.180108 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.220608 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.330467 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.328904 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.349803 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.349515 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.466026 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.466224 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.465289 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.461329 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.569833 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.579678 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.577190 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.608975 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.696140 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.730631 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.733962 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.750709 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.837822 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.844895 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.859723 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.869661 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:43.955473 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:43.955508 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:43.974697 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:43.994726 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.086096 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.099586 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.113051 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.099606 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.217361 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.225658 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.206142 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.234880 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.366076 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.381288 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.329851 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.303888 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.495972 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.530099 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.558177 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.640435 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0111 23:03:44.750172 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0111 23:03:44.755552 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0111 23:03:44.760118 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-1]: I0111 23:03:44.758373 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.775667 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.778073 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.775778 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.779527 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:44.864164 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:44.877098 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:44.880253 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:44.889789 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.033525 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.044201 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.059633 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.060631 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.190214 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.213502 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.209936 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.226111 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.379614 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.429365 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.436557 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.423487 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:45.789345 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:45.816072 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:45.820211 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:45.874403 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.129642 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.120414 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.127145 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.137177 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.234757 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.238713 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.249571 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.350576 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.524798 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.511001 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.531679 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.520399 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:46.829920 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:46.840196 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:46.849372 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:46.940325 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.089866 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.099981 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.105787 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.109392 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.349662 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.346007 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.359536 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.369181 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.539859 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.539855 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.549456 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.570036 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:47.679829 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:47.689801 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:47.679887 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:47.722022 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.030182 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.040240 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.039419 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.050097 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I0111 23:03:48.234364 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-1]: I0111 23:03:48.233057 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-0]: I0111 23:03:48.235363 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:epoch 5 finished [worker-3]: I0111 23:03:48.235918 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.259684 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.259535 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.259858 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.318994 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.388763 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.384312 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.399730 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.420902 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.539971 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.549731 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.562679 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.610249 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.699621 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.703996 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.719607 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.719627 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:48.811597 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:48.824172 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:48.829773 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:48.839534 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:49.054402 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:49.049595 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:49.050661 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:49.059591 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:49.169508 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:49.209492 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:49.199662 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:49.230028 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:49.349765 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:49.350111 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:49.359790 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:49.365936 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:49.459878 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:49.462562 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:49.489719 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:49.516026 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:49.719777 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:49.729724 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:49.740239 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:49.737401 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:49.812647 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:49.816302 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:49.830084 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:49.830226 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:49.922392 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:49.939968 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:49.947599 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:49.959621 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:50.068724 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:50.075965 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:50.090025 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:50.115661 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:50.209723 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:50.234082 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:50.239670 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:50.279953 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:50.388606 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:50.395562 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:50.399761 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:50.419973 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-3]: I0111 23:03:50.532508 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I0111 23:03:50.535386 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-0]: I0111 23:03:50.534046 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-2]: I0111 23:03:50.565389 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:50.560441 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:50.569612 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:50.589385 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:50.593203 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:50.859774 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:50.888341 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:50.872454 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:50.926177 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:51.139623 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:51.169625 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:51.171217 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:51.281038 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:51.471910 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:51.469466 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:51.490474 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:51.506913 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:51.682596 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:51.701389 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:51.731013 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:51.796492 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:51.969507 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:52.120664 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:52.160394 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:52.342997 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:52.480695 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:52.499805 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:52.500566 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:52.547819 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:52.700026 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:52.716086 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:52.739340 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:52.778182 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:52.940337 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:52.961874 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:53.121520 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:53.141896 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:53.246310 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:53.273792 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:53.291016 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:53.289013 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:53.599854 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:53.609524 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:53.612137 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:53.712163 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:53.899760 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:53.919900 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:53.935517 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:53.909851 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:54.361046 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:54.377391 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:54.400113 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:54.424755 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:54.648363 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:54.659796 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:54.679669 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:54.777225 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:54.946491 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:54.959873 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:54.969660 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:55.070876 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I0111 23:03:55.282860 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-2]: I0111 23:03:55.285745 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0111 23:03:55.287060 281473065645088 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I0111 23:03:55.295382 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-3]: I0111 23:03:55.294007 281473065645088 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I0111 23:03:55.301768 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0111 23:03:55.303153 281473065645088 failure_handler_test.py:245] Training finished. [worker-0]: I0111 23:03:55.296913 281473065645088 failure_handler_test.py:245] Training finished. I0111 23:03:56.561444 281473280668704 multi_process_runner.py:646] worker-0 exit code: 0 I0111 23:03:56.561738 281473280668704 multi_process_runner.py:646] worker-1 exit code: 0 I0111 23:03:56.561890 281473280668704 multi_process_runner.py:646] worker-2 exit code: 0 I0111 23:03:56.562025 281473280668704 multi_process_runner.py:646] worker-3 exit code: 0 I0111 23:03:56.566452 281473280668704 multi_process_runner.py:662] Joining log reading threads. I0111 23:03:56.566796 281473280668704 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 29.79s I0111 23:03:56.840706 281473280668704 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 29.79s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 23852 I0111 23:03:56.844634 281473280668704 test_util.py:3999] Using local port 23852 INFO:tensorflow:Using local port 23851 I0111 23:03:56.846640 281473280668704 test_util.py:3999] Using local port 23851 INFO:tensorflow:Using local port 23850 I0111 23:03:56.848970 281473280668704 test_util.py:3999] Using local port 23850 INFO:tensorflow:Using local port 23849 I0111 23:03:56.850677 281473280668704 test_util.py:3999] Using local port 23849 INFO:tensorflow:Cluster starting. I0111 23:03:56.998744 281473280668704 failure_handler_test.py:297] Cluster starting. [worker-1]: I0111 23:03:57.091041 281473065645088 multi_process_runner.py:840] Subprocess with PID 1792276 (worker, 1) is now being started. [worker-2]: I0111 23:03:57.091688 281473065645088 multi_process_runner.py:840] Subprocess with PID 1792405 (worker, 2) is now being started. [worker-0]: I0111 23:03:57.087472 281473065645088 multi_process_runner.py:840] Subprocess with PID 1792271 (worker, 0) is now being started. [worker-2]: I0111 23:03:57.092121 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23852", "localhost:23851", "localhost:23850", "localhost:23849"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: I0111 23:03:57.087946 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23852", "localhost:23851", "localhost:23850", "localhost:23849"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0111 23:03:57.091501 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23852", "localhost:23851", "localhost:23850", "localhost:23849"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0111 23:03:57.139799 281473065645088 multi_process_runner.py:840] Subprocess with PID 1792788 (worker, 3) is now being started. [worker-3]: I0111 23:03:57.140262 281473065645088 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23852", "localhost:23851", "localhost:23850", "localhost:23849"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2024-01-11 23:03:57.164274: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23850 [worker-1]: 2024-01-11 23:03:57.193521: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23851 [worker-0]: 2024-01-11 23:03:57.196619: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23852 [worker-3]: 2024-01-11 23:03:57.239226: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23849 [worker-0]: 2024-01-11 23:03:57.256294: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 8496856920511531189 [worker-0]: 2024-01-11 23:03:57.256350: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2024-01-11 23:03:57.256361: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:0 [worker-0]: /job:worker/replica:0/task:1 [worker-0]: /job:worker/replica:0/task:3 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-2]: I0000 00:00:1705014237.256599 1792405 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:57.261723: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 7940621997983469791 [worker-0]: 2024-01-11 23:03:57.261776: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-0]: 2024-01-11 23:03:57.261788: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:1 [worker-0]: /job:worker/replica:0/task:3 [worker-0]: 2024-01-11 23:03:57.261832: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 1020923567443813732 [worker-0]: 2024-01-11 23:03:57.261855: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-0]: 2024-01-11 23:03:57.261866: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:1 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1705014237.263147 1792271 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: I0000 00:00:1705014237.262862 1792788 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2024-01-11 23:03:57.264554: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 4060289384964209647 [worker-0]: 2024-01-11 23:03:57.264598: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0000 00:00:1705014237.275216 1792276 coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0111 23:03:57.279247 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0111 23:03:57.278416 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0111 23:03:57.297235 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0111 23:03:57.317422 281473065645088 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0111 23:03:57.359040 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0111 23:03:57.359614 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0111 23:03:57.359854 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0111 23:03:57.375768 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0111 23:03:57.376372 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0111 23:03:57.376617 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0111 23:03:57.413817 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0111 23:03:57.414399 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0111 23:03:57.414643 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0111 23:03:57.510641 281473065645088 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0111 23:03:57.512113 281473065645088 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0111 23:03:57.512377 281473065645088 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23852', 'localhost:23851', 'localhost:23850', 'localhost:23849']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0111 23:03:57.602151 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0111 23:03:57.602995 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:57.601687 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0111 23:03:57.603251 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0111 23:03:57.603548 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0111 23:03:57.603733 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0111 23:03:57.615411 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0111 23:03:57.635579 281473065645088 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0111 23:03:57.635367 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-1]: I0111 23:03:57.635583 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0111 23:03:57.635743 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: I0111 23:03:57.635915 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: Instructions for updating: [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0111 23:03:57.636221 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: W0111 23:03:57.636059 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-0]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0111 23:03:57.636426 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0111 23:03:57.636249 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-3]: I0111 23:03:57.636859 281473065645088 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0111 23:03:57.637125 281473065645088 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0111 23:03:57.637432 281473065645088 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0111 23:03:57.637619 281473065645088 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.766551 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.811318 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.817859 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.823259 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.893955 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:57.894375 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.894341 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.893965 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:57.976637 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:57.977131 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:57.982621 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.019883 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0111 23:04:01.109558 281473280668704 failure_handler_test.py:302] sending sigterm [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 18.46s I0111 23:04:15.301441 281473280668704 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 18.46s ====================================================================== ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='manager', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process ---------------------------------------------------------------------- Ran 4 tests in 59.493s FAILED (errors=1) [worker-0]: I0111 23:03:58.347736 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.347737 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.347987 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.347882 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.404674 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.404587 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefb7b00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: I0111 23:03:58.405283 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefba660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0111 23:03:58.404824 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W0111 23:03:58.453431 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefba660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefb91c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.463509 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0111 23:03:58.453292 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefb7b00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb98a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffedefb9440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:58.453575 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefb91c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0111 23:03:58.453653 281473065645088 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffedefb9440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0111 23:03:58.510664 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb98a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.462140 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.462275 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.462275 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.520526 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb8220> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0111 23:03:58.510811 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb8220> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb4360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffedefb8b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0111 23:03:58.510532 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb4360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0111 23:03:58.510845 281473065645088 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffedefb8b80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0111 23:03:58.520523 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.520623 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.629616 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.520469 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.630129 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.686623 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.632003 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.687835 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.686578 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.743210 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.630121 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.743229 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.744708 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.799220 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.687241 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.799217 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.799793 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.857376 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.743891 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.857319 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.859267 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.916569 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.799837 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.916822 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:58.972292 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:58.972294 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.917197 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.859266 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.027908 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:58.973457 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.027901 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:58.917724 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.077629 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: I0111 23:03:59.028275 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.077701 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.086254 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: I0111 23:03:58.972618 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.086257 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.077360 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.141709 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.028223 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.087296 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.141715 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.077517 281473065645088 failure_handler_test.py:195] epoch 0 finished [worker-1]: I0111 23:03:59.196239 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.086552 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.142143 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.196684 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.196238 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.254115 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.252457 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.311838 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.142147 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.252453 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.370015 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.311842 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.426550 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.370010 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.482409 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.426309 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.313457 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.196688 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.538703 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.371187 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.254114 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.426684 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.313459 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.482402 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.483087 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.598400 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.538707 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.370408 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.654229 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.598347 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.426502 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.711686 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.654273 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.483087 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.773292 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.712317 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.540951 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.540954 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.832612 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.600080 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.773272 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.600081 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.892791 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.654860 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.832530 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.654860 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.713875 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:03:59.943070 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.892770 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.775429 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.713863 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: I0111 23:03:59.951098 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.943010 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-0]: I0111 23:03:59.834966 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.775444 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.010394 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.894910 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:03:59.950906 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.835110 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.070346 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.942952 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.010411 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.894911 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:03:59.952716 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.129923 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-1]: I0111 23:04:00.070475 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.942777 281473065645088 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.129888 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.012755 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.189423 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.189410 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:03:59.952550 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.072330 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.247419 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.304740 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.132024 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.247874 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.012761 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.190968 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.304783 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.072340 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.359800 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.249866 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.359784 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.132023 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.415678 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.306018 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.415726 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.190953 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.472368 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.360866 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.472417 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.249975 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.416945 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.305918 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.528285 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.474159 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.359957 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.585203 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.528413 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.415409 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.639724 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.530430 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.472440 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.586252 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.695409 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.527990 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.751622 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.584835 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-0]: I0111 23:04:00.640666 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.801402 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.639429 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.697094 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.809097 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.695162 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.753193 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.868765 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.751285 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.801408 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-2]: I0111 23:04:00.926306 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.801052 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-0]: I0111 23:04:00.811440 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:00.983402 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.870905 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.808974 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.585140 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.928022 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.868536 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.039738 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.639733 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:00.985059 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.695395 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.040704 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.751549 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.095742 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.926064 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.153595 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.097283 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:00.983545 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.801324 281473065645088 failure_handler_test.py:195] epoch 2 finished [worker-3]: I0111 23:04:01.039462 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.154904 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.809021 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.095494 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.868690 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.211295 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.153353 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.926190 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.267055 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:00.983402 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.209925 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.323509 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.210087 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.039641 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.265617 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.266278 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.381026 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.322408 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.322206 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.095656 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.378695 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.378477 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.440923 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.153723 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.438327 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.438507 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.210063 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.500129 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.497735 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.265800 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.497860 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.559765 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.556977 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.557210 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.322378 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.378684 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.617782 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.620256 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: I0111 23:04:01.438380 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.617484 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.670356 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.678512 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.738234 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.670376 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-2]: I0111 23:04:01.795365 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.497997 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.670035 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.681240 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.855524 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.557199 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.677836 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.914719 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.617706 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-3]: I0111 23:04:01.737954 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.740252 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:01.974296 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.670307 281473065645088 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.034625 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.094409 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.795446 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.797847 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.678010 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.857497 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.154579 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.738179 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.855357 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.917100 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.213923 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.914480 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.795376 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:01.976800 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.272506 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.855449 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:01.974113 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.037177 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.331617 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.034439 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.096904 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.156859 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.391090 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.914656 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.216104 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.094123 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.450040 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.274640 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.509016 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.154288 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.333599 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.561717 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-3]: I0111 23:04:02.213683 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.392701 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.569467 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.272274 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.452335 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.626431 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.331410 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.511293 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: I0111 23:04:02.684861 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.390363 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.561744 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.743989 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.449841 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.571924 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.802069 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.628177 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.509204 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-2]: I0111 23:04:02.859604 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.687225 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.561398 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.918076 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.569349 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.746590 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:02.975271 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.802654 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.626193 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.031075 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.684593 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.861228 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.088021 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.743728 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.919953 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.143507 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:02.976270 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.801383 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.199054 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.033348 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.859608 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.255318 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.088586 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.918141 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:01.974326 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.310550 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:02.975026 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.034553 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.144580 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.030833 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.094342 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.199826 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.369838 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-3]: I0111 23:04:03.087838 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.154462 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.255721 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.418749 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.213829 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.143203 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.426549 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.312783 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.272823 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.480352 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.371917 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.198738 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.331569 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.255065 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.533220 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.418845 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.310137 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.586024 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.427523 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.390626 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.369548 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.639577 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-1]: I0111 23:04:02.449972 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.480789 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.418399 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.694072 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.509019 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.533714 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.427770 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.747563 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.561648 281473065645088 failure_handler_test.py:195] epoch 4 finished [worker-3]: I0111 23:04:03.480731 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.586434 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.800390 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.569367 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.533892 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.640373 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.854003 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.626344 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.694823 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.586364 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.912434 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.684986 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.748096 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.640377 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:03.969676 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.743869 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.800853 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.801649 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.026634 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.853470 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.859602 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.694781 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.082136 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.918082 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.748070 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.138387 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:02.975154 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.800864 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.193148 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.031038 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.853435 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.238467 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-1]: I0111 23:04:03.087948 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.246182 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.910804 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.143414 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.298437 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:03.968293 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.199178 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.350821 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.024917 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.255213 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.403334 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.081163 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.310362 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.459796 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.137180 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.369894 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.516046 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-3]: I0111 23:04:04.192421 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.418654 281473065645088 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.571621 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.238147 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-1]: I0111 23:04:03.426549 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.624188 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.246328 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.480186 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.677174 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.298093 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.533121 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.729268 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.351223 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.585912 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.784108 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.403070 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.639556 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.840842 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.458536 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.693909 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.900403 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.747534 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.515059 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:04.956096 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.800326 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:05.011669 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.571209 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.854004 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:05.059513 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-3]: I0111 23:04:04.624311 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.912555 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.676841 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0111 23:04:05.060571 281473065645088 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:03.969660 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.728979 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.026437 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.782383 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.082138 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.839272 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.138391 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.900410 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.193148 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:04.956104 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.238417 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-3]: I0111 23:04:05.011671 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-1]: I0111 23:04:04.246234 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:05.059214 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Training finished. [worker-1]: I0111 23:04:04.298437 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0111 23:04:05.060431 281473065645088 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.350856 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.403299 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.459795 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.910917 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:03.968402 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.024964 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.081147 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.137286 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.192484 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-0]: I0111 23:04:04.238296 281473065645088 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.246337 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.298137 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.351289 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.403131 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.458524 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.515088 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.571344 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.624311 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.676893 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.729087 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.516046 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.571599 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.782473 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.624137 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.839310 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.677182 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.901357 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.729335 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:04.955015 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.784108 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:05.010771 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-1]: I0111 23:04:04.840935 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:05.059228 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0111 23:04:04.900403 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0111 23:04:05.060093 281473065645088 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:04.956098 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0111 23:04:05.011664 281473065645088 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I0111 23:04:05.059473 281473065645088 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0111 23:04:05.060665 281473065645088 failure_handler_test.py:245] Training finished. ================================================================================ ==================== Test output for //tensorflow/core/kernels:batch_kernels_test: [==========] Running 6 tests from 2 test suites. [----------] Global test environment set-up. [----------] 2 tests from Params/BatchFunctionKernelTest [ RUN ] Params/BatchFunctionKernelTest.EnableAdaptiveScheduler/0 [ OK ] Params/BatchFunctionKernelTest.EnableAdaptiveScheduler/0 (37 ms) [ RUN ] Params/BatchFunctionKernelTest.EnableAdaptiveScheduler/1 [ OK ] Params/BatchFunctionKernelTest.EnableAdaptiveScheduler/1 (11 ms) [----------] 2 tests from Params/BatchFunctionKernelTest (48 ms total) [----------] 4 tests from BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest [ RUN ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmup/0 2024-01-11 23:08:19.621584: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:19.765902: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:19.765989: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:19.766165: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] [ OK ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmup/0 (1845 ms) [ RUN ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmup/1 2024-01-11 23:08:21.469426: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:21.487153: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:21.498447: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] tensorflow/core/kernels/batch_kernels_test.cc:205: Failure Value of: test.RunOpKernel().ok() Actual: true Expected: false 2024-01-11 23:08:21.712972: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [8] is not compatible with expected shape [2]. [[{{node o}}]] [ FAILED ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmup/1, where GetParam() = true (1928 ms) [ RUN ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmupAutoBatch/0 2024-01-11 23:08:21.925247: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:23.685737: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:23.786229: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:23.886574: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:23.987035: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.087064: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.187161: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.295205: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.395728: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.495864: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.596255: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.696978: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.799616: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:24.900031: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:25.000344: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:25.100673: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] [ OK ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmupAutoBatch/0 (20159 ms) [ RUN ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmupAutoBatch/1 2024-01-11 23:08:42.075398: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:43.817378: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:43.917439: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.017576: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.118839: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.225286: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.329144: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.433512: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.535253: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.665237: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.765921: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.866241: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:44.975270: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:45.075521: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:45.175726: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] 2024-01-11 23:08:45.275998: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Shape of tensor x [4] is not compatible with expected shape [2]. [[{{node o}}]] [ OK ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmupAutoBatch/1 (19566 ms) [----------] 4 tests from BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest (54548 ms total) [----------] Global test environment tear-down [==========] 6 tests from 2 test suites ran. (54601 ms total) [ PASSED ] 5 tests. [ FAILED ] 1 test, listed below: [ FAILED ] BatchFunctionKernelParallelWarmupTestSuite/BatchFunctionKernelParallelWarmupTest.ParallelWarmup/1, where GetParam() = true 1 FAILED TEST ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 34.6s //tensorflow/c:c_api_function_test PASSED in 38.2s //tensorflow/c:c_api_test_cpu PASSED in 42.3s //tensorflow/c:c_test PASSED in 42.8s //tensorflow/c:env_test_cpu PASSED in 29.1s //tensorflow/c:kernels_test_cpu PASSED in 41.9s //tensorflow/c:ops_test PASSED in 32.9s //tensorflow/c:tf_status_helper_test PASSED in 0.2s //tensorflow/c:while_loop_test PASSED in 39.8s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 44.7s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 31.7s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 35.3s //tensorflow/c/eager:c_api_test_cpu PASSED in 37.5s //tensorflow/c/eager:custom_device_test PASSED in 38.2s //tensorflow/c/eager:dlpack_test_cpu PASSED in 34.3s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 40.0s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 35.1s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 30.8s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.1s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.3s //tensorflow/c/experimental/grappler:grappler_test PASSED in 33.4s //tensorflow/c/experimental/next_pluggable_device:tensor_pjrt_buffer_util_test PASSED in 8.4s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 1.4s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.5s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 2.4s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 21.2s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 16.9s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 24.5s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 16.7s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 20.7s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 18.5s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 21.5s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 38.0s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.1s //tensorflow/c/kernels:bitcast_op_test PASSED in 1.6s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 0.7s //tensorflow/c/kernels:summary_op_test PASSED in 0.6s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.4s //tensorflow/cc:cc_op_gen_test PASSED in 0.5s //tensorflow/cc:client_client_session_test PASSED in 9.6s //tensorflow/cc:coordinator_test PASSED in 4.1s //tensorflow/cc:framework_cc_ops_test PASSED in 2.1s //tensorflow/cc:framework_gradient_checker_test PASSED in 2.9s //tensorflow/cc:framework_gradients_test PASSED in 5.3s //tensorflow/cc:framework_scope_test PASSED in 1.3s //tensorflow/cc:framework_while_gradients_test PASSED in 3.3s //tensorflow/cc:gradients_array_grad_test PASSED in 10.6s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 3.1s //tensorflow/cc:gradients_functional_grad_test PASSED in 2.4s //tensorflow/cc:gradients_image_grad_test PASSED in 54.9s //tensorflow/cc:gradients_linalg_grad_test PASSED in 2.3s //tensorflow/cc:gradients_manip_grad_test PASSED in 2.0s //tensorflow/cc:gradients_math_grad_test PASSED in 5.8s //tensorflow/cc:gradients_nn_grad_test PASSED in 4.1s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.0s //tensorflow/cc:ops_const_op_test PASSED in 0.5s //tensorflow/cc:ops_while_loop_test PASSED in 2.6s //tensorflow/cc:queue_runner_test PASSED in 12.1s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.4s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 35.4s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.1s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 38.1s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 35.2s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 35.6s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.3s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.3s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.3s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.1s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.1s //tensorflow/cc/saved_model:fingerprinting_chunked_test PASSED in 0.3s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.0s //tensorflow/cc/saved_model:fingerprinting_utils_test PASSED in 0.6s //tensorflow/cc/saved_model:metrics_test PASSED in 0.3s //tensorflow/cc/saved_model:reader_test PASSED in 0.1s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 6.0s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 21.9s //tensorflow/cc/saved_model:util_test PASSED in 0.5s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 32.3s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 16.6s //tensorflow/compiler/aot:codegen_test PASSED in 37.7s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 21.7s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 11.7s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 9.0s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 15.6s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 20.7s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 6.0s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 20.7s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 29.3s //tensorflow/compiler/jit:device_util_test PASSED in 7.7s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 1.0s //tensorflow/compiler/jit:node_matchers_test PASSED in 0.4s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 12.7s //tensorflow/compiler/jit:shape_inference_test PASSED in 1.0s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 28.6s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 11.7s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 6.4s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 10.1s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 32.3s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 36.1s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 31.8s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 23.1s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite:offset_buffer_test PASSED in 0.1s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 1.9s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 8.3s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 1.5s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.1s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 14.7s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 14.3s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 16.3s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:call_xla_module_to_stablehlo.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:compose-uniform-quantized-type.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize_hlo.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tfl_legalize_hlo.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:tfl_legalize_hlo_custom_call.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfold_splat_constant_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:uniform-quantized-stablehlo-to-tfl.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tensorlist.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/lite/tests:optimize_batch_matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 3.6s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:reduce-type-precision.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants_offset.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op_offset.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:mix_tflite_stablehlo.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_const.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_custom_call.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:variable.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_tensorlist_reserve.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:deduplicate_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:u16_quant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variable.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/common:attrs_and_constraints_test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/common:lift_as_function_call_test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test PASSED in 16.3s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_types_test PASSED in 22.6s //tensorflow/compiler/mlir/quantization/stablehlo:math_utils_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/stablehlo:stablehlo_type_utils_test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/stablehlo:tf_type_utils_test PASSED in 35.4s //tensorflow/compiler/mlir/quantization/stablehlo:uniform_quantized_types_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/stablehlo/cc:export_test PASSED in 10.8s //tensorflow/compiler/mlir/quantization/stablehlo/cc:graph_def_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo/cc:io_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/stablehlo/cc:pre_calibration_test PASSED in 16.5s //tensorflow/compiler/mlir/quantization/stablehlo/cc/calibration:assign_ids_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/stablehlo/python:quantize_model_test PASSED in 98.6s //tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 3.2s //tensorflow/compiler/mlir/quantization/stablehlo/tests:stablehlo_op_quant_spec_test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibration_algorithm_test PASSED in 37.7s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibration_statistics_collector_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 27.2s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/cc:constant_fold_test PASSED in 4.9s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 8.1s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/ops:tf_op_quant_spec_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/ops:tf_quantize_op_test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 58.4s //tensorflow/compiler/mlir/quantization/tensorflow/python:py_function_lib_py_test PASSED in 38.9s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 72.2s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 20.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:add_dump_tensor_op.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:add_quantization_unit_loc.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_xla_op_to_tf_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_hashtable_ops_as_args.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla_selective_quantization.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_duplicate_resource_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 12.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:propagate_quantize_type.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composit_functions_debugging.mlir.test PASSED in 5.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 4.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_weights.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 17.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_uniform_attribute_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 39.8s //tensorflow/compiler/mlir/stablehlo:stablehlo_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 8.1s //tensorflow/compiler/mlir/tensorflow:call_graph_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:data_dumper_logger_config_test PASSED in 7.9s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 18.5s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 7.0s //tensorflow/compiler/mlir/tensorflow:xla_rewrite_util_test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:colocate_tpu_copy_with_dynamic_shape.mlir.test PASSED in 3.2s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:embedding_program_key.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:embedding_sequencing.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:move_tpu_compile_to_front.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:print.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 16.0s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_asset_sinking.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tfrt_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 12.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:verify_for_export.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_outline_entry_functions.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite_v2.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:xla_sharding_util_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_validate_iputs.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:convert_mhlo_quant_to_int.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 11.4s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 3.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 5.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graph:convert_tensor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 3.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/transforms:verify_no_outside_compilation_markers_pass_test PASSED in 21.5s //tensorflow/compiler/mlir/tensorflow/transforms/host_runtime:lower_cluster_to_runtime_ops_test PASSED in 19.5s //tensorflow/compiler/mlir/tensorflow/transforms/host_runtime:tpu_metadata_utils_test PASSED in 16.3s //tensorflow/compiler/mlir/tensorflow/translate:tf_mlir_translate_registration_test PASSED in 18.5s //tensorflow/compiler/mlir/tf2xla/api/v1:cluster_tf_test PASSED in 28.2s //tensorflow/compiler/mlir/tf2xla/api/v1:compile_mlir_util_test PASSED in 5.9s //tensorflow/compiler/mlir/tf2xla/api/v1:compile_tf_graph_test PASSED in 0.3s //tensorflow/compiler/mlir/tf2xla/api/v1:tf_dialect_to_executor_test PASSED in 24.3s //tensorflow/compiler/mlir/tf2xla/api/v2:cluster_tf_test PASSED in 29.5s //tensorflow/compiler/mlir/tf2xla/api/v2:legalize_tf_test PASSED in 36.0s //tensorflow/compiler/mlir/tf2xla/api/v2:tf_dialect_to_executor_test PASSED in 20.6s //tensorflow/compiler/mlir/tf2xla/internal:clustering_bridge_passes_test PASSED in 8.1s //tensorflow/compiler/mlir/tf2xla/internal:compilation_timer_test PASSED in 0.4s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_mlir_test PASSED in 23.0s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_to_hlo_test PASSED in 21.4s //tensorflow/compiler/mlir/tf2xla/internal:logging_hooks_test PASSED in 20.0s //tensorflow/compiler/mlir/tf2xla/internal:mlir_pass_instrumentation_test PASSED in 6.5s //tensorflow/compiler/mlir/tf2xla/internal:test_matchers_test PASSED in 14.9s //tensorflow/compiler/mlir/tf2xla/internal/inference:inference_metrics_pass_test PASSED in 21.4s //tensorflow/compiler/mlir/tf2xla/internal/passes:tpu_cluster_formation_test PASSED in 20.4s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_clustering_pass_test PASSED in 19.0s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_clustering_pass_test.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_input_dialect_to_executor_pass_test.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/internal/utils:dialect_detection_utils_test PASSED in 0.5s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-quant.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 8.5s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/transforms:legalization_op_config_test PASSED in 34.3s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 21.1s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 21.8s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 3.9s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 14.9s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 26.4s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 24.6s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 9.0s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 23.5s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 40.7s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 35.6s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 52.2s //tensorflow/compiler/mlir/tfrt/tests:batch_function_fallback_resource_variable_as_captured_tensor.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:batch_function_lowering.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tfrt/tests:convert_ref_variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:cross_device_transfer.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests:deduplicate_if_results.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:fuse_tpu_compile_and_execute_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops_mlrt.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:optimize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:remove_device_attribute.mlir.test PASSED in 7.8s //tensorflow/compiler/mlir/tfrt/tests:runtime_lowering_gpu.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:runtime_lowering_tpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:sink_in_invariant_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_fallback.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_lowering.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:xla_rewrite.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/analysis:cost_analysis.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/analysis:tensor_array_side_effect_analysis.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/analysis:update_op_cost_in_tfrt_mlir_test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/ifrt:rewrite_cluster_to_ifrt_call.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tfrt/tests/ir:fallback_opt.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/ir:tfrt_fallback_util_test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/mlrt:assign_op_key.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/mlrt:async_while.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/mlrt:fuse_mlrt_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/mlrt:inline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/mlrt:parallelization.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tf_to_mlrt.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tpu_conversions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:while_to_map_fn.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:attributes.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:basic.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate_failed.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:const_tensor.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:control_flow.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:decompose_resource_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:derived_attrs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:device_conversion.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:errors.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_canonicalization.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_inline.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes_multiple_callers.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_use_fallback_tensor.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:insert_fallback_tensor_copy.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:merge_tf_if_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:optimize_tf_control_flow_side_effect.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:remove_tf_if_const_args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:reorder_assert.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:side_effects.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline_refvar.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:whileop.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/translate/mlrt:mlir_to_bytecode_test PASSED in 0.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:func_to_jit_invocations.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 6.4s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-stateful.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 2.0s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 18.7s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 26.9s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 13.8s //tensorflow/compiler/tests:adam_test_cpu PASSED in 16.9s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 12.1s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 65.3s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 37.5s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 9.8s //tensorflow/compiler/tests:case_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 9.6s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 12.3s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 20.2s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 15.0s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 31.8s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 21.3s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 11.5s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 11.0s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 11.7s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 12.9s //tensorflow/compiler/tests:cond_test_cpu PASSED in 22.3s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 13.3s //tensorflow/compiler/tests:const_test_cpu PASSED in 16.3s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 19.2s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 29.5s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 21.0s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 16.7s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 12.4s //tensorflow/compiler/tests:eager_test_cpu PASSED in 20.4s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 20.0s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 13.0s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 13.2s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 20.5s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 15.0s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 17.7s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 23.0s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 13.0s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 13.5s //tensorflow/compiler/tests:function_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 12.2s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 23.2s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 14.3s //tensorflow/compiler/tests:gather_test_cpu PASSED in 31.6s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 58.2s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 16.0s //tensorflow/compiler/tests:jit_test_cpu PASSED in 68.8s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 31.7s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 14.8s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 15.0s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 31.6s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 31.8s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 17.9s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 14.8s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 26.2s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 34.4s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 15.3s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 13.1s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 14.5s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 21.0s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 11.9s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 32.5s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 11.2s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 9.7s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 13.3s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 9.9s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 11.6s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 14.3s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 62.6s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 13.8s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 13.8s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 21.1s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 13.6s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 17.1s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 23.5s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 41.9s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 17.6s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 12.2s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 26.7s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 24.4s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 16.7s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 37.8s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 30.9s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 25.6s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 10.0s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 13.3s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 16.7s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 14.0s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 29.9s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 12.0s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 25.0s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 20.7s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 22.1s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 21.2s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 10.8s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 31.3s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 19.3s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 12.5s //tensorflow/compiler/tests:while_test_cpu PASSED in 14.6s //tensorflow/compiler/tests:xla_call_module_no_platform_check_test_cpu PASSED in 15.4s //tensorflow/compiler/tests:xla_call_module_no_shape_assertions_check_test_cpu PASSED in 12.7s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 16.6s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 10.8s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 18.2s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 18.1s //tensorflow/compiler/tests:xla_dump_to_test_cpu PASSED in 11.7s //tensorflow/compiler/tests:xla_dump_to_test_cpu_mlir_bridge_test PASSED in 10.5s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 37.6s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 36.2s //tensorflow/compiler/tests:xla_test_test PASSED in 28.0s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 17.9s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.1s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 1.0s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 1.3s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 26.8s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 6.5s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 0.6s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 5.8s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 1.9s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 2.2s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 9.9s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 18.7s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.9s //tensorflow/compiler/tf2xla:type_util_test PASSED in 1.0s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 19.2s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 19.6s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 6.3s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 1.8s //tensorflow/core:@local_tsl__tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 12.8s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 77.3s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.7s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.1s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.8s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.2s //tensorflow/core/api_def:api_test PASSED in 3.6s //tensorflow/core/api_def:update_api_def_test PASSED in 0.4s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 0.5s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.7s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 0.9s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 1.1s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 5.2s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 0.9s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.5s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 3.0s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.1s //tensorflow/core/common_runtime:device_mgr_test PASSED in 1.0s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.5s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 1.2s //tensorflow/core/common_runtime:device_set_test PASSED in 1.1s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 2.3s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 1.8s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 0.9s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 0.8s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.6s //tensorflow/core/common_runtime:executor_test PASSED in 2.1s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 1.2s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 1.5s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 0.9s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 0.9s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.0s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 2.5s //tensorflow/core/common_runtime:graph_runner_test PASSED in 0.7s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 2.8s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 0.5s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.7s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 0.5s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 2.3s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 3.9s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 4.0s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 3.3s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 1.9s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.8s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.4s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.4s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.1s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.3s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 1.0s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 5.9s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 0.7s //tensorflow/core/common_runtime:pending_counts_test PASSED in 1.2s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 3.6s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 1.3s //tensorflow/core/common_runtime:placer_test PASSED in 1.5s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.8s //tensorflow/core/common_runtime:process_util_test PASSED in 0.7s //tensorflow/core/common_runtime:quantize_training_test PASSED in 2.6s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.2s //tensorflow/core/common_runtime:replicate_constants_pass_test PASSED in 1.2s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.5s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.7s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.7s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 2.6s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 5.1s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 4.4s //tensorflow/core/common_runtime:session_test PASSED in 1.9s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 1.8s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 1.1s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 0.9s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.5s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 32.2s //tensorflow/core/common_runtime/eager:context_test PASSED in 15.6s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 13.6s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 13.8s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 0.9s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 13.3s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 9.8s //tensorflow/core/common_runtime/eager:execute_test PASSED in 50.1s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 1.0s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 13.7s //tensorflow/core/common_runtime/eager:placement_test PASSED in 12.9s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 13.5s //tensorflow/core/common_runtime/eager:summary_optimizer_test PASSED in 0.5s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 9.8s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 10.0s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.1s //tensorflow/core/common_runtime/gpu:gpu_serving_device_selector_test PASSED in 0.1s //tensorflow/core/common_runtime/next_pluggable_device:c_plugin_coordination_service_agent_test PASSED in 4.9s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 34.1s //tensorflow/core/common_runtime/next_pluggable_device/c:tf_rendezvous_c_api_test PASSED in 0.1s //tensorflow/core/config:flags_py_test PASSED in 27.1s //tensorflow/core/config:flags_test PASSED in 0.1s //tensorflow/core/data:compression_utils_test PASSED in 2.1s //tensorflow/core/data:dataset_utils_test PASSED in 1.0s //tensorflow/core/data:hash_utils_test PASSED in 1.0s //tensorflow/core/data:metric_utils_test PASSED in 5.8s //tensorflow/core/data:name_utils_test PASSED in 0.1s //tensorflow/core/data:rewrite_utils_test PASSED in 1.4s //tensorflow/core/data:serialization_utils_test PASSED in 0.9s //tensorflow/core/data:snapshot_utils_test PASSED in 0.6s //tensorflow/core/data:split_utils_test PASSED in 0.5s //tensorflow/core/data:standalone_save_restore_test PASSED in 2.6s //tensorflow/core/data:standalone_test PASSED in 4.9s //tensorflow/core/data:tfdataz_metrics_test PASSED in 1.9s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 1.8s //tensorflow/core/data:utils_test PASSED in 0.1s //tensorflow/core/data/service:auto_scaler_test PASSED in 0.2s //tensorflow/core/data/service:common_test PASSED in 0.1s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.8s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.7s //tensorflow/core/data/service:data_service_test PASSED in 12.9s //tensorflow/core/data/service:data_transfer_test PASSED in 0.9s //tensorflow/core/data/service:dataset_store_test PASSED in 0.8s //tensorflow/core/data/service:dispatcher_client_test PASSED in 3.9s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.6s //tensorflow/core/data/service:graph_rewriters_test PASSED in 0.8s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 2.8s //tensorflow/core/data/service:grpc_util_test PASSED in 0.6s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 2.5s //tensorflow/core/data/service:journal_test PASSED in 0.7s //tensorflow/core/data/service:logging_utils_test PASSED in 0.1s //tensorflow/core/data/service:split_provider_test PASSED in 2.4s //tensorflow/core/data/service:task_runner_test PASSED in 7.1s //tensorflow/core/data/service:test_util_test PASSED in 2.3s //tensorflow/core/data/service:url_test PASSED in 0.7s //tensorflow/core/data/service:utils_test PASSED in 0.4s //tensorflow/core/data/service:validate_utils_test PASSED in 0.1s //tensorflow/core/data/service:worker_client_test PASSED in 16.8s //tensorflow/core/data/service:worker_impl_test PASSED in 3.0s //tensorflow/core/data/service/client:data_service_client_test PASSED in 3.0s //tensorflow/core/data/service/client:utils_test PASSED in 2.8s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.9s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 19.0s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.5s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 1.4s //tensorflow/core/data/service/snapshot:snapshot_chunk_provider_test PASSED in 1.3s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 1.8s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 1.7s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 15.5s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 4.2s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.2s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.7s //tensorflow/core/distributed_runtime:call_options_test PASSED in 1.1s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 4.1s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 0.9s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.5s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 1.5s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.1s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.1s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.6s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.6s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 0.9s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.1s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.3s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 30.3s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 20.8s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 40.6s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 47.4s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 30.9s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 2.7s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 1.6s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 0.6s //tensorflow/core/example:example_parser_configuration_test PASSED in 1.4s //tensorflow/core/example:feature_util_test PASSED in 0.2s //tensorflow/core/framework:allocator_test PASSED in 4.9s //tensorflow/core/framework:attr_value_util_test PASSED in 0.9s //tensorflow/core/framework:batch_util_test PASSED in 17.9s //tensorflow/core/framework:bfloat16_test PASSED in 1.5s //tensorflow/core/framework:common_shape_fns_test PASSED in 0.9s //tensorflow/core/framework:dataset_test PASSED in 1.0s //tensorflow/core/framework:device_base_test PASSED in 0.9s //tensorflow/core/framework:disable_jit_test PASSED in 0.9s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.4s //tensorflow/core/framework:framework_op_segment_test PASSED in 1.7s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.1s //tensorflow/core/framework:framework_run_handler_test PASSED in 1.6s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.1s //tensorflow/core/framework:full_type_inference_util_test PASSED in 0.9s //tensorflow/core/framework:full_type_util_test PASSED in 1.1s //tensorflow/core/framework:function_test PASSED in 1.0s //tensorflow/core/framework:graph_def_util_test PASSED in 1.9s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 0.8s //tensorflow/core/framework:kernel_def_builder_test PASSED in 1.1s //tensorflow/core/framework:kernel_def_util_test PASSED in 1.3s //tensorflow/core/framework:memory_types_test PASSED in 1.3s //tensorflow/core/framework:model_test PASSED in 1.4s //tensorflow/core/framework:node_def_builder_test PASSED in 1.3s //tensorflow/core/framework:node_def_util_test PASSED in 1.2s //tensorflow/core/framework:node_properties_test PASSED in 1.0s //tensorflow/core/framework:op_compatibility_test PASSED in 1.1s //tensorflow/core/framework:op_def_builder_test PASSED in 0.9s //tensorflow/core/framework:op_def_util_test PASSED in 0.9s //tensorflow/core/framework:op_kernel_test PASSED in 1.1s //tensorflow/core/framework:op_registration_test PASSED in 1.4s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 1.7s //tensorflow/core/framework:rendezvous_test PASSED in 3.3s //tensorflow/core/framework:resource_handle_test PASSED in 0.5s //tensorflow/core/framework:resource_mgr_test PASSED in 2.7s //tensorflow/core/framework:resource_op_kernel_test PASSED in 1.6s //tensorflow/core/framework:shape_inference_test PASSED in 0.8s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 1.4s //tensorflow/core/framework:tensor_matcher_test PASSED in 1.5s //tensorflow/core/framework:tensor_shape_test PASSED in 9.0s //tensorflow/core/framework:tensor_slice_test PASSED in 1.3s //tensorflow/core/framework:tensor_test PASSED in 31.9s //tensorflow/core/framework:tensor_testutil_test PASSED in 0.8s //tensorflow/core/framework:tensor_util_test PASSED in 0.9s //tensorflow/core/framework:tracking_allocator_test PASSED in 0.7s //tensorflow/core/framework:types_test PASSED in 0.7s //tensorflow/core/framework:variant_op_registry_test PASSED in 20.4s //tensorflow/core/framework:variant_test PASSED in 1.2s //tensorflow/core/framework/registration:registration_test PASSED in 0.6s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 12.2s //tensorflow/core/function/capture:capture_container_test PASSED in 12.0s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 21.5s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 25.6s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 10.4s //tensorflow/core/function/polymorphism:function_type_test PASSED in 8.5s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 16.6s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 48.7s //tensorflow/core/function/trace_type:custom_nest_trace_type_test PASSED in 26.7s //tensorflow/core/function/trace_type:default_types_test PASSED in 24.3s //tensorflow/core/function/trace_type:serialization_test PASSED in 14.6s //tensorflow/core/function/trace_type:trace_type_test PASSED in 12.3s //tensorflow/core/graph:algorithm_test PASSED in 0.9s //tensorflow/core/graph:collective_order_test PASSED in 0.5s //tensorflow/core/graph:control_flow_test PASSED in 1.0s //tensorflow/core/graph:costmodel_test PASSED in 1.8s //tensorflow/core/graph:edgeset_test PASSED in 1.0s //tensorflow/core/graph:graph_debug_info_builder_test PASSED in 1.1s //tensorflow/core/graph:graph_def_builder_test PASSED in 1.0s //tensorflow/core/graph:graph_partition_test PASSED in 1.3s //tensorflow/core/graph:graph_test PASSED in 1.1s //tensorflow/core/graph:node_builder_test PASSED in 2.2s //tensorflow/core/graph:optimizer_cse_test PASSED in 1.0s //tensorflow/core/graph:subgraph_test PASSED in 1.2s //tensorflow/core/graph:tensor_id_test PASSED in 0.7s //tensorflow/core/graph:validate_test PASSED in 1.5s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 6.3s //tensorflow/core/graph/regularization:util_test PASSED in 0.1s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.1s //tensorflow/core/grappler:graph_view_test PASSED in 1.9s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 1.1s //tensorflow/core/grappler:grappler_item_test PASSED in 1.6s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 2.4s //tensorflow/core/grappler:utils_test PASSED in 2.1s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 23.4s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 1.6s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 1.6s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.3s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 2.0s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 12.4s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.1s //tensorflow/core/grappler/costs:utils_test PASSED in 4.5s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.4s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.8s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 1.7s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 1.8s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 2.2s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 3.1s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 1.5s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 13.8s //tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu PASSED in 4.2s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 3.6s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 3.2s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 2.5s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 3.4s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 6.2s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.3s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 2.8s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 2.0s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 8.0s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 2.5s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 2.8s //tensorflow/core/grappler/optimizers:remapper_test_cpu PASSED in 3.0s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 1.6s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:inject_io_prefetch_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 12.0s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:remove_compression_map_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 2.2s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.5s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 1.2s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.5s //tensorflow/core/grappler/utils:frame_test PASSED in 0.7s //tensorflow/core/grappler/utils:functions_test PASSED in 1.6s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.8s //tensorflow/core/grappler/utils:graph_view_test PASSED in 2.3s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 10.0s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.7s //tensorflow/core/grappler/utils:scc_test PASSED in 1.5s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.1s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.8s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.9s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.5s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.7s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.7s //tensorflow/core/ir:interfaces_test PASSED in 0.3s //tensorflow/core/ir:ops_test PASSED in 0.2s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.3s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.4s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.3s //tensorflow/core/ir:utility_test PASSED in 0.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 1.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 1.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 1.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 2.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 1.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 1.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 1.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.4s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 1.4s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 1.8s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 1.2s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 0.9s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 1.6s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.9s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.6s //tensorflow/core/ir/types:dialect_test PASSED in 0.3s //tensorflow/core/kernels:as_string_op_test PASSED in 0.9s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.5s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 1.2s //tensorflow/core/kernels:bias_op_test PASSED in 0.7s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 3.0s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 0.6s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.5s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.5s //tensorflow/core/kernels:concat_op_test PASSED in 1.2s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:control_flow_ops_test PASSED in 6.8s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 7.1s //tensorflow/core/kernels:count_ops_test PASSED in 0.8s //tensorflow/core/kernels:cross_op_test PASSED in 1.8s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 1.0s //tensorflow/core/kernels:debug_ops_test PASSED in 1.1s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.7s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.7s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.7s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.1s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.2s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 0.7s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.4s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.1s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.4s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.3s //tensorflow/core/kernels:encode_wav_op_test PASSED in 2.6s //tensorflow/core/kernels:fingerprint_op_test PASSED in 0.6s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 17.5s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 1.3s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.5s //tensorflow/core/kernels:identity_op_test PASSED in 1.3s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 1.1s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.4s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 0.8s //tensorflow/core/kernels:logging_ops_test PASSED in 1.7s //tensorflow/core/kernels:lookup_ops_test PASSED in 0.7s //tensorflow/core/kernels:loss_test PASSED in 0.1s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 5.0s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 0.7s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.2s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.8s //tensorflow/core/kernels:mfcc_test PASSED in 0.1s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.7s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.7s //tensorflow/core/kernels:ops_util_test PASSED in 0.2s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:parse_tensor_test PASSED in 1.0s //tensorflow/core/kernels:quantization_utils_test PASSED in 1.1s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 1.0s //tensorflow/core/kernels:quantize_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.5s //tensorflow/core/kernels:quantized_add_op_test PASSED in 2.0s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.8s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 2.9s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.5s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 1.0s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 12.1s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 0.8s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 1.1s //tensorflow/core/kernels:ragged_range_op_test PASSED in 0.9s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.8s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 0.8s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 2.7s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 0.5s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.5s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:range_sampler_test PASSED in 0.4s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.6s //tensorflow/core/kernels:requantization_range_op_test PASSED in 0.6s //tensorflow/core/kernels:requantize_op_test PASSED in 0.7s //tensorflow/core/kernels:resource_ops_test PASSED in 0.5s //tensorflow/core/kernels:restore_op_test PASSED in 1.4s //tensorflow/core/kernels:restore_v2_op_test PASSED in 1.1s //tensorflow/core/kernels:reverse_op_test PASSED in 0.5s //tensorflow/core/kernels:roll_op_test PASSED in 0.6s //tensorflow/core/kernels:save_op_test PASSED in 1.1s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.7s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:scatter_op_test PASSED in 0.6s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 6.5s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.5s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 3.0s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.7s //tensorflow/core/kernels:sequence_ops_test PASSED in 1.0s //tensorflow/core/kernels:shape_ops_test PASSED in 0.7s //tensorflow/core/kernels:slice_op_test PASSED in 0.6s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.9s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.9s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 1.3s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 1.3s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.4s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 3.1s //tensorflow/core/kernels:spectrogram_test PASSED in 0.2s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.8s //tensorflow/core/kernels:string_format_op_test PASSED in 0.7s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 2.9s //tensorflow/core/kernels:string_split_op_test PASSED in 0.9s //tensorflow/core/kernels:substr_op_test PASSED in 1.1s //tensorflow/core/kernels:summary_audio_op_test PASSED in 1.5s //tensorflow/core/kernels:summary_image_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_op_test PASSED in 1.5s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.6s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.4s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.1s //tensorflow/core/kernels:tensor_map_test PASSED in 0.1s //tensorflow/core/kernels:training_ops_test PASSED in 0.6s //tensorflow/core/kernels:transpose_util_test PASSED in 0.4s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 2.8s //tensorflow/core/kernels:unique_op_test PASSED in 0.4s //tensorflow/core/kernels:variable_ops_test PASSED in 1.9s //tensorflow/core/kernels:while_op_test PASSED in 2.7s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 1.0s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.5s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 1.8s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 2.2s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 18.4s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.1s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 0.8s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:get_options_op_test PASSED in 1.2s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 3.3s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 1.1s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.5s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 2.3s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.4s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 2.4s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 2.7s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 2.3s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 3.5s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 2.1s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 2.2s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.6s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 1.7s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 0.7s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.8s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 1.0s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 1.1s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.8s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 0.6s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.0s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 5.0s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 2.4s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 0.4s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 2.3s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 12.2s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.8s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.5s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.5s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.2s //tensorflow/core/kernels/sparse:kernels_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 0.7s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 0.6s //tensorflow/core/lib/db:sqlite_test PASSED in 0.1s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 1.7s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 1.8s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 0.8s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.6s //tensorflow/core/ops:ops_math_grad_test PASSED in 3.4s //tensorflow/core/ops:ops_tests PASSED in 10.8s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.5s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.1s //tensorflow/core/platform:env_test PASSED in 2.6s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 17.9s //tensorflow/core/platform:platform_strings_test PASSED in 0.2s //tensorflow/core/platform:ram_file_system_test PASSED in 19.2s //tensorflow/core/platform:resource_loader_test PASSED in 0.1s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.2s //tensorflow/core/platform:vmodule_test PASSED in 0.5s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.4s //tensorflow/core/profiler/convert:dcn_analysis_test PASSED in 0.2s //tensorflow/core/profiler/convert:dcn_utils_test PASSED in 0.1s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_combiner_test PASSED in 0.9s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.5s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.4s //tensorflow/core/profiler/convert:repository_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_dcn_collective_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.5s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.2s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.1s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 0.8s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 1.2s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 0.5s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 0.8s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 0.6s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.4s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.2s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.2s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.1s //tensorflow/core/runtime_fallback/util:type_util_test PASSED in 0.1s //tensorflow/core/summary:schema_test PASSED in 0.1s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.3s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.1s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 8.0s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 12.9s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 7.8s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.2s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.6s //tensorflow/core/tfrt/graph_executor:config_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/attribute:attribute_test PASSED in 0.6s //tensorflow/core/tfrt/mlrt/bytecode:bytecode_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/bytecode:executable_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:function_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:kernel_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:span_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:context_test PASSED in 0.4s //tensorflow/core/tfrt/mlrt/interpreter:future_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:interpreter_test PASSED in 0.4s //tensorflow/core/tfrt/mlrt/interpreter:register_span_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:value_test PASSED in 0.1s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_concurrent_work_queue_test PASSED in 1.3s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_test PASSED in 0.9s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_util_test PASSED in 0.3s //tensorflow/core/tfrt/runtime:tf_threadpool_concurrent_work_queue_test PASSED in 1.2s //tensorflow/core/tfrt/runtime:work_queue_interface_test PASSED in 0.1s //tensorflow/core/tfrt/utils:graph_partition_test PASSED in 2.1s //tensorflow/core/transforms:eval_utils_test PASSED in 1.2s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.3s //tensorflow/core/util:bcast_test PASSED in 1.1s //tensorflow/core/util:command_line_flags_test PASSED in 1.0s //tensorflow/core/util:debug_data_dumper_test PASSED in 1.5s //tensorflow/core/util:debug_events_writer_test PASSED in 0.4s //tensorflow/core/util:dump_graph_test PASSED in 2.0s //tensorflow/core/util:equal_graph_def_test PASSED in 0.8s //tensorflow/core/util:events_writer_test PASSED in 2.9s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.9s //tensorflow/core/util:example_proto_helper_test PASSED in 0.7s //tensorflow/core/util:exec_on_stall_test PASSED in 2.1s //tensorflow/core/util:fake_clock_env_test PASSED in 2.9s //tensorflow/core/util:incremental_barrier_test PASSED in 0.2s //tensorflow/core/util:matmul_bcast_test PASSED in 1.3s //tensorflow/core/util:memmapped_file_system_test PASSED in 0.8s //tensorflow/core/util:mkl_heuristics_test PASSED in 0.1s //tensorflow/core/util:overflow_test PASSED in 0.1s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 2.2s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.5s //tensorflow/core/util:reffed_status_callback_test PASSED in 1.0s //tensorflow/core/util:reporter_test PASSED in 1.2s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.9s //tensorflow/core/util:semver_test PASSED in 0.9s //tensorflow/core/util:stat_summarizer_test PASSED in 0.9s //tensorflow/core/util:strided_slice_op_test PASSED in 1.0s //tensorflow/core/util:tensor_format_test PASSED in 0.7s //tensorflow/core/util:tensor_slice_reader_test PASSED in 0.9s //tensorflow/core/util:tensor_slice_set_test PASSED in 0.9s //tensorflow/core/util:tensor_slice_util_test PASSED in 1.8s //tensorflow/core/util:tensor_slice_writer_test PASSED in 1.6s //tensorflow/core/util:work_sharder_test PASSED in 1.6s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.3s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.7s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.5s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.6s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.1s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 39.8s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:decompose_controlflow.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 3.9s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_collective_type_lowering.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 1.7s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 2.1s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:spmd_fft.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 2.3s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 1.7s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 1.0s //tensorflow/dtensor/python/tests:api_test PASSED in 32.1s //tensorflow/dtensor/python/tests:array_ops_test_cpu PASSED in 28.2s //tensorflow/dtensor/python/tests:cache_test_cpu PASSED in 26.8s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 36.6s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 19.9s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 9.4s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 58.1s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 21.6s //tensorflow/dtensor/python/tests:mesh_util_test_cpu PASSED in 14.6s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 18.7s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 12.5s //tensorflow/dtensor/python/tests:variable_test_cpu PASSED in 24.2s //tensorflow/dtensor/tests:dtensor_operation_test PASSED in 37.1s //tensorflow/dtensor/tests:executable_manager_test PASSED in 30.1s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.3s //tensorflow/dtensor/tests:slice_util_test PASSED in 0.2s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 8.3s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.4s //tensorflow/examples/adding_an_op:fact_test PASSED in 29.2s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 24.3s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 40.2s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 41.3s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 25.6s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 22.4s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 30.1s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 54.8s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 24.4s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 25.7s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 2.3s //tensorflow/examples/speech_commands:models_test PASSED in 40.3s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.9s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.1s //tensorflow/js:ts_op_gen_test PASSED in 0.2s //tensorflow/python/autograph/converters:asserts_test PASSED in 9.7s //tensorflow/python/autograph/converters:break_statements_test PASSED in 9.5s //tensorflow/python/autograph/converters:call_trees_test PASSED in 12.9s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 31.0s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 11.4s //tensorflow/python/autograph/converters:control_flow_test PASSED in 22.2s //tensorflow/python/autograph/converters:directives_test PASSED in 24.0s //tensorflow/python/autograph/converters:functions_test PASSED in 10.4s //tensorflow/python/autograph/converters:lists_test PASSED in 11.6s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 11.8s //tensorflow/python/autograph/converters:return_statements_test PASSED in 11.4s //tensorflow/python/autograph/converters:slices_test PASSED in 10.7s //tensorflow/python/autograph/converters:variables_test PASSED in 10.2s //tensorflow/python/autograph/core:converter_test PASSED in 11.3s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 11.0s //tensorflow/python/autograph/impl:api_test PASSED in 35.6s //tensorflow/python/autograph/impl:conversion_test PASSED in 9.1s //tensorflow/python/autograph/lang:special_functions_test PASSED in 10.8s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 9.7s //tensorflow/python/autograph/operators:control_flow_test PASSED in 24.1s //tensorflow/python/autograph/operators:data_structures_test PASSED in 10.6s //tensorflow/python/autograph/operators:exceptions_test PASSED in 15.3s //tensorflow/python/autograph/operators:logical_test PASSED in 18.9s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 29.8s //tensorflow/python/autograph/operators:slices_test PASSED in 12.8s //tensorflow/python/autograph/operators:variables_test PASSED in 9.5s //tensorflow/python/autograph/pyct:anno_test PASSED in 16.5s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 9.4s //tensorflow/python/autograph/pyct:cache_test PASSED in 10.9s //tensorflow/python/autograph/pyct:cfg_test PASSED in 13.5s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 9.2s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 12.0s //tensorflow/python/autograph/pyct:loader_test PASSED in 11.6s //tensorflow/python/autograph/pyct:naming_test PASSED in 20.2s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 9.2s //tensorflow/python/autograph/pyct:parser_test PASSED in 9.9s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 10.4s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 13.5s //tensorflow/python/autograph/pyct:templates_test PASSED in 9.7s //tensorflow/python/autograph/pyct:transformer_test PASSED in 9.7s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 10.5s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 8.8s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 13.0s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 9.3s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 9.3s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 10.7s //tensorflow/python/autograph/tests:assertion_test PASSED in 44.9s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 33.1s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 39.6s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 27.9s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 24.2s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 45.8s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 81.2s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 41.3s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 60.9s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 33.0s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 49.2s //tensorflow/python/autograph/tests:datasets_test PASSED in 29.6s //tensorflow/python/autograph/tests:early_return_test PASSED in 45.1s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 23.2s //tensorflow/python/autograph/tests:generator_test PASSED in 22.0s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 30.8s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 95.9s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 27.1s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 33.0s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 35.4s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 55.3s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 32.3s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 42.0s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 57.6s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 31.5s //tensorflow/python/autograph/utils:context_managers_test PASSED in 12.5s //tensorflow/python/autograph/utils:misc_test PASSED in 13.2s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 11.3s //tensorflow/python/autograph/utils:tensors_test PASSED in 17.9s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 15.0s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 32.2s //tensorflow/python/checkpoint:checkpoint_test PASSED in 32.6s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 12.6s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 31.4s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 19.6s //tensorflow/python/checkpoint:restore_test PASSED in 22.8s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 9.7s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 15.9s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 11.2s //tensorflow/python/checkpoint:trackable_view_test PASSED in 25.8s //tensorflow/python/checkpoint/sharding:sharding_policies_test PASSED in 14.0s //tensorflow/python/checkpoint/sharding:sharding_util_test PASSED in 11.6s //tensorflow/python/client:device_lib_test_cpu PASSED in 9.6s //tensorflow/python/client:events_writer_test PASSED in 11.8s //tensorflow/python/client:session_list_devices_test PASSED in 10.8s //tensorflow/python/client:session_partial_run_test PASSED in 29.5s //tensorflow/python/client:timeline_test_cpu PASSED in 20.1s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 14.0s //tensorflow/python/compat:compat_test PASSED in 24.5s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 12.5s //tensorflow/python/compiler/mlir:mlir_test PASSED in 11.5s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 17.2s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 21.3s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 12.1s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 18.0s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 11.4s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 26.8s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 13.5s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 20.7s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 15.7s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 13.8s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 11.9s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 11.9s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 11.2s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 13.9s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 14.3s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 15.4s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 13.8s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 22.0s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 24.9s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 12.6s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 11.5s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 15.2s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 12.6s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 16.6s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 30.8s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 12.5s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 28.5s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 28.4s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 14.4s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 24.4s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 20.8s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 22.1s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 24.3s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 25.5s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 69.6s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 19.5s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 27.1s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 12.3s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 58.3s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 11.0s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 44.6s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 12.6s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 22.0s //tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 13.1s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 16.6s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 25.8s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 27.5s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 15.2s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 23.7s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 11.0s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 14.7s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 39.2s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 56.1s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 12.6s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 55.8s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 10.6s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 21.5s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 200.2s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 14.7s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 13.2s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 20.2s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 10.3s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 21.7s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 24.3s //tensorflow/python/data/kernel_tests:cache_test PASSED in 68.1s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 36.6s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 23.1s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 33.2s //tensorflow/python/data/kernel_tests:counter_test PASSED in 44.0s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 12.1s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 31.5s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 28.7s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 11.1s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 66.8s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 24.9s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 19.0s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 20.8s //tensorflow/python/data/kernel_tests:io_test PASSED in 52.7s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 42.9s //tensorflow/python/data/kernel_tests:len_test PASSED in 10.8s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 19.5s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 20.5s //tensorflow/python/data/kernel_tests:options_test PASSED in 14.0s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 20.8s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 41.0s //tensorflow/python/data/kernel_tests:random_test PASSED in 30.4s //tensorflow/python/data/kernel_tests:range_test PASSED in 45.5s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 23.9s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 30.1s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 66.7s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 26.9s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 36.4s //tensorflow/python/data/util:convert_test PASSED in 11.2s //tensorflow/python/data/util:nest_test PASSED in 19.5s //tensorflow/python/data/util:options_test PASSED in 30.4s //tensorflow/python/data/util:random_seed_test PASSED in 11.3s //tensorflow/python/data/util:sparse_test PASSED in 32.9s //tensorflow/python/data/util:structure_test PASSED in 33.4s //tensorflow/python/data/util:traverse_test PASSED in 10.8s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 12.5s //tensorflow/python/debug/cli:cli_config_test PASSED in 9.6s //tensorflow/python/debug/cli:cli_shared_test PASSED in 15.2s //tensorflow/python/debug/cli:command_parser_test PASSED in 11.7s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 9.3s //tensorflow/python/debug/cli:evaluator_test PASSED in 9.9s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 13.5s //tensorflow/python/debug/cli:readline_ui_test PASSED in 11.0s //tensorflow/python/debug/cli:tensor_format_test PASSED in 9.9s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 14.4s //tensorflow/python/debug/lib:common_test PASSED in 9.5s //tensorflow/python/debug/lib:debug_data_test PASSED in 9.8s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 10.9s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 11.5s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 11.4s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 12.2s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 9.1s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 15.7s //tensorflow/python/debug/lib:debug_utils_test PASSED in 10.2s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 20.0s //tensorflow/python/debug/lib:profiling_test PASSED in 16.5s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 19.7s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 10.0s //tensorflow/python/debug/lib:source_utils_test PASSED in 14.5s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 30.2s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 10.2s //tensorflow/python/debug/wrappers:framework_test PASSED in 12.9s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 11.9s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 18.2s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 13.4s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 15.4s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 13.8s //tensorflow/python/distribute:collective_util_test PASSED in 26.8s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 38.5s //tensorflow/python/distribute:combinations_test_cpu PASSED in 28.8s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 11.4s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 15.6s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 18.2s //tensorflow/python/distribute:device_util_test_cpu PASSED in 64.3s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 18.9s //tensorflow/python/distribute:distribute_lib_test PASSED in 25.9s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 11.8s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 28.9s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 34.7s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 35.4s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 48.6s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 25.5s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 31.2s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 27.0s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 45.6s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 11.8s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 25.2s //tensorflow/python/distribute:multi_worker_util_test PASSED in 9.9s //tensorflow/python/distribute:numpy_dataset_test PASSED in 13.6s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 32.8s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 11.0s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 43.1s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 46.7s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 27.3s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 24.9s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 13.9s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 15.8s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 12.0s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 22.1s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 13.7s //tensorflow/python/distribute:sharded_variable_test PASSED in 24.2s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 24.6s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 66.5s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 11.9s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 23.6s //tensorflow/python/distribute:test_util_test_cpu PASSED in 20.6s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 21.2s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 19.3s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 16.7s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 13.7s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 13.7s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 11.4s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 10.5s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 9.6s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 9.8s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 10.6s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 11.3s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 13.1s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 64.1s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 17.6s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 35.0s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 17.8s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 68.6s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 21.5s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 50.9s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 78.5s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 21.6s //tensorflow/python/eager:backprop_test_cpu PASSED in 159.9s //tensorflow/python/eager:cancellation_test_cpu PASSED in 9.4s //tensorflow/python/eager:context_test_cpu PASSED in 11.8s //tensorflow/python/eager:core_test_cpu PASSED in 22.5s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 42.6s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 10.4s //tensorflow/python/eager:lift_to_graph_test PASSED in 12.5s //tensorflow/python/eager:monitoring_test_cpu PASSED in 12.9s //tensorflow/python/eager:ops_test_cpu PASSED in 9.1s //tensorflow/python/eager:profiler_client_test PASSED in 14.1s //tensorflow/python/eager:profiler_test_cpu PASSED in 9.4s //tensorflow/python/eager:pywrap_tfe_test PASSED in 17.7s //tensorflow/python/eager:record_test PASSED in 10.5s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 11.7s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 13.9s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 215.6s //tensorflow/python/eager:tensor_test_cpu PASSED in 14.7s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 13.1s //tensorflow/python/eager:wrap_function_test PASSED in 26.2s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 9.7s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 11.5s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 11.5s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 10.9s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 13.4s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 13.5s //tensorflow/python/eager/polymorphic_function:concrete_function_test_cpu PASSED in 18.0s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 14.3s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 11.1s //tensorflow/python/eager/polymorphic_function:tracing_compilation_test PASSED in 18.8s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 13.8s //tensorflow/python/feature_column:serialization_test PASSED in 20.5s //tensorflow/python/framework:auto_control_deps_test PASSED in 36.4s //tensorflow/python/framework:c_api_util_test PASSED in 10.8s //tensorflow/python/framework:common_shapes_test PASSED in 11.4s //tensorflow/python/framework:composite_tensor_test PASSED in 10.7s //tensorflow/python/framework:config_test_2gpu PASSED in 16.3s //tensorflow/python/framework:config_test_cpu PASSED in 30.5s //tensorflow/python/framework:constant_op_test PASSED in 11.4s //tensorflow/python/framework:device_spec_test PASSED in 10.0s //tensorflow/python/framework:device_test PASSED in 22.5s //tensorflow/python/framework:dtypes_test PASSED in 26.2s //tensorflow/python/framework:error_interpolation_test PASSED in 10.9s //tensorflow/python/framework:errors_test PASSED in 10.5s //tensorflow/python/framework:extension_type_field_test PASSED in 10.2s //tensorflow/python/framework:extension_type_test PASSED in 20.9s //tensorflow/python/framework:file_system_test PASSED in 10.4s //tensorflow/python/framework:flexible_dtypes_test PASSED in 118.9s //tensorflow/python/framework:function_def_to_graph_test PASSED in 11.5s //tensorflow/python/framework:graph_util_test PASSED in 11.6s //tensorflow/python/framework:immutable_dict_test PASSED in 9.4s //tensorflow/python/framework:importer_test PASSED in 12.5s //tensorflow/python/framework:indexed_slices_test PASSED in 9.4s //tensorflow/python/framework:kernels_test PASSED in 10.0s //tensorflow/python/framework:meta_graph_test PASSED in 11.9s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 10.3s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.6s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 5.7s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 17.2s //tensorflow/python/framework:op_def_library_test PASSED in 9.4s //tensorflow/python/framework:op_def_util_test PASSED in 9.4s //tensorflow/python/framework:ops_enable_eager_test PASSED in 4.2s //tensorflow/python/framework:ops_test PASSED in 23.6s //tensorflow/python/framework:proto_test PASSED in 9.3s //tensorflow/python/framework:py_context_manager_test PASSED in 8.8s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 11.7s //tensorflow/python/framework:python_api_info_test PASSED in 9.6s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 11.3s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 7.1s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.1s //tensorflow/python/framework:python_op_gen_test PASSED in 0.2s //tensorflow/python/framework:python_tensor_converter_test PASSED in 11.1s //tensorflow/python/framework:random_seed_test PASSED in 14.1s //tensorflow/python/framework:registry_test PASSED in 14.0s //tensorflow/python/framework:smart_cond_test PASSED in 13.1s //tensorflow/python/framework:sparse_tensor_test PASSED in 14.7s //tensorflow/python/framework:subscribe_test PASSED in 10.1s //tensorflow/python/framework:tensor_shape_test PASSED in 11.2s //tensorflow/python/framework:tensor_test PASSED in 10.6s //tensorflow/python/framework:tensor_util_test PASSED in 13.5s //tensorflow/python/framework:test_combinations_test PASSED in 27.1s //tensorflow/python/framework:test_util_test_cpu PASSED in 20.9s //tensorflow/python/framework:tf2_test PASSED in 10.9s //tensorflow/python/framework:traceable_stack_test PASSED in 9.7s //tensorflow/python/framework:type_spec_test PASSED in 10.3s //tensorflow/python/framework:versions_test PASSED in 17.0s //tensorflow/python/framework:weak_tensor_test PASSED in 13.9s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 32.3s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 18.2s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 16.3s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 10.5s //tensorflow/python/grappler:cost_analyzer_test PASSED in 11.8s //tensorflow/python/grappler:datasets_test PASSED in 13.0s //tensorflow/python/grappler:item_test PASSED in 8.9s //tensorflow/python/grappler:memory_optimizer_test PASSED in 46.2s //tensorflow/python/grappler:model_analyzer_test PASSED in 11.4s //tensorflow/python/grappler:remapper_test_cpu PASSED in 11.6s //tensorflow/python/grappler:tf_optimizer_test PASSED in 10.9s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 20.8s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 31.6s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 10.1s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 22.6s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 11.1s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 14.3s //tensorflow/python/kernel_tests:template_test PASSED in 14.6s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 15.9s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 12.3s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 18.5s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 20.8s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 40.2s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 9.3s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 13.0s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 30.6s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 27.3s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 20.0s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 21.9s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 16.1s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 17.0s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 25.2s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 22.9s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 65.9s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 13.2s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 16.0s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 29.7s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 96.9s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 73.7s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 11.2s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 19.6s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 13.3s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 17.6s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 17.2s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 16.8s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 30.9s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 27.8s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 17.1s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 43.2s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 19.0s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 11.4s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 17.7s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 17.5s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 19.6s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 32.8s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 18.9s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 69.2s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 34.1s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 37.3s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 23.9s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 22.5s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 11.0s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 12.4s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 16.2s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 9.4s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 11.6s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 20.8s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 26.0s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 14.5s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 31.1s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 10.7s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 12.6s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 26.2s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 13.3s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 26.2s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 11.2s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 32.0s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 64.3s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 30.4s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 24.4s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 19.1s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 19.9s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 19.2s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 27.8s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 17.6s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 13.5s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 29.4s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 28.8s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 20.4s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 145.6s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 14.3s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 17.8s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 11.6s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 78.0s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 54.9s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 14.4s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 16.8s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 31.4s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 20.7s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 123.3s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/proto:decode_proto_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 10.5s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 18.4s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 10.1s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 13.1s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 17.9s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 28.8s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 15.3s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 18.9s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 15.8s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 10.3s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 30.0s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 226.6s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 19.0s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 10.9s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 16.1s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 36.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 11.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 63.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 9.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 16.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 38.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 14.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 14.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 11.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 27.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 40.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 18.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 10.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 81.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 12.1s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 13.5s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 9.7s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 11.9s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 15.1s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 9.9s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 8.3s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 12.9s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 15.7s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 13.3s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 23.7s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 10.0s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 19.8s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 10.8s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 34.1s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 12.4s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 12.1s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 31.1s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 21.0s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 8.6s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 14.7s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 73.2s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 58.1s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 12.8s //tensorflow/python/lib/io:file_io_test PASSED in 14.3s //tensorflow/python/lib/io:tf_record_test PASSED in 13.7s //tensorflow/python/module:module_test PASSED in 14.3s //tensorflow/python/ops:array_grad_test_cpu PASSED in 17.0s //tensorflow/python/ops:array_ops_shape_test PASSED in 9.4s //tensorflow/python/ops:array_ops_test PASSED in 13.4s //tensorflow/python/ops:autograph_ops_test PASSED in 13.0s //tensorflow/python/ops:bincount_ops_test_cpu PASSED in 13.7s //tensorflow/python/ops:bitwise_ops_test_cpu PASSED in 11.9s //tensorflow/python/ops:clip_ops_test PASSED in 13.5s //tensorflow/python/ops:clustering_ops_test PASSED in 26.9s //tensorflow/python/ops:collective_ops_gpu_test_cpu PASSED in 15.2s //tensorflow/python/ops:collective_ops_test PASSED in 16.9s //tensorflow/python/ops:collective_ops_xla_test PASSED in 13.2s //tensorflow/python/ops:compiled_collective_ops_gpu_test_2gpu PASSED in 13.9s //tensorflow/python/ops:compiled_collective_ops_gpu_test_cpu PASSED in 19.8s //tensorflow/python/ops:control_flow_v2_enable_test PASSED in 13.5s //tensorflow/python/ops:control_flow_v2_toggles_test PASSED in 11.6s //tensorflow/python/ops:dequantize_op_test PASSED in 12.5s //tensorflow/python/ops:embedding_ops_test_cpu PASSED in 10.0s //tensorflow/python/ops:factory_ops_test_cpu PASSED in 11.5s //tensorflow/python/ops:functional_ops_test PASSED in 11.1s //tensorflow/python/ops:gradient_checker_v2_test_cpu PASSED in 38.1s //tensorflow/python/ops:gradients_test_cpu PASSED in 20.5s //tensorflow/python/ops:init_ops_test_cpu PASSED in 11.7s //tensorflow/python/ops:init_ops_v2_test_cpu PASSED in 13.1s //tensorflow/python/ops:lookup_ops_async_checkpoint_test PASSED in 11.6s //tensorflow/python/ops:math_grad_test_cpu PASSED in 22.9s //tensorflow/python/ops:math_ops_linspace_test_cpu PASSED in 10.1s //tensorflow/python/ops:math_ops_test_cpu PASSED in 50.4s //tensorflow/python/ops:nn_grad_test_cpu PASSED in 11.6s //tensorflow/python/ops:nn_loss_scaling_utilities_test PASSED in 16.6s //tensorflow/python/ops:nn_test_cpu PASSED in 61.5s //tensorflow/python/ops:nn_xent_test_cpu PASSED in 13.7s //tensorflow/python/ops:op_selector_test PASSED in 9.4s //tensorflow/python/ops:quantized_conv_ops_test PASSED in 14.6s //tensorflow/python/ops:quantized_ops_test PASSED in 9.9s //tensorflow/python/ops:raw_ops_test_cpu PASSED in 10.2s //tensorflow/python/ops:rnn_grad_test_cpu PASSED in 12.4s //tensorflow/python/ops:script_ops_test PASSED in 9.8s //tensorflow/python/ops:sort_ops_test PASSED in 10.7s //tensorflow/python/ops:sparse_bincount_ops_test_cpu PASSED in 14.6s //tensorflow/python/ops:sparse_ops_test PASSED in 25.7s //tensorflow/python/ops:tensor_array_ops_test PASSED in 12.9s //tensorflow/python/ops:variable_spec_test PASSED in 13.3s //tensorflow/python/ops:weak_tensor_array_ops_test PASSED in 25.6s //tensorflow/python/ops:weak_tensor_constant_op_test PASSED in 18.0s //tensorflow/python/ops:weak_tensor_image_ops_test PASSED in 9.8s //tensorflow/python/ops:weak_tensor_math_ops_test PASSED in 24.6s //tensorflow/python/ops:weak_tensor_nn_test_cpu PASSED in 17.9s //tensorflow/python/ops:weak_tensor_np_array_ops_test PASSED in 41.1s //tensorflow/python/ops:weak_tensor_np_math_ops_test PASSED in 12.2s //tensorflow/python/ops:weak_tensor_ops_test PASSED in 110.9s //tensorflow/python/ops/losses:util_test PASSED in 15.8s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 15.8s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 97.3s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 10.3s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 17.0s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 60.9s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 12.7s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 34.8s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 57.7s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 8.8s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 25.1s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 34.8s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 45.1s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 16.3s //tensorflow/python/ops/parallel_for:pfor_test PASSED in 11.1s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 68.3s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 16.5s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 52.8s //tensorflow/python/ops/ragged:ragged_bincount_ops_test_cpu PASSED in 12.7s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 16.3s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 17.0s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 21.6s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 11.4s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 10.1s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 24.8s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 170.8s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 19.6s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 10.1s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 25.8s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 20.1s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 12.5s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 11.8s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 22.8s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 14.1s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 13.3s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 22.3s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 15.8s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 37.4s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 28.1s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 15.4s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 23.4s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 30.8s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 24.7s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 13.7s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 11.1s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 56.2s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 46.7s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 12.3s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 11.5s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 9.9s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 27.3s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 22.9s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 20.9s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 67.7s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 23.5s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 14.8s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 12.6s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 67.4s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 58.8s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 24.7s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 58.7s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 25.0s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 51.5s //tensorflow/python/ops/ragged:row_partition_test PASSED in 31.7s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 10.4s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 16.3s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 45.5s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 63.6s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 14.7s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 49.5s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 17.5s //tensorflow/python/platform:benchmark_test PASSED in 10.3s //tensorflow/python/platform:build_info_test PASSED in 9.6s //tensorflow/python/platform:resource_loader_test PASSED in 3.7s //tensorflow/python/profiler:pprof_profiler_test PASSED in 16.9s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 25.5s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 17.4s //tensorflow/python/profiler:profiler_test_cpu PASSED in 26.9s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 12.6s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 9.9s //tensorflow/python/profiler:tfprof_logger_test PASSED in 9.5s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 10.9s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 15.4s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 17.8s //tensorflow/python/saved_model:fingerprinting_test PASSED in 11.9s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 18.3s //tensorflow/python/saved_model:loader_test PASSED in 24.1s //tensorflow/python/saved_model:method_name_updater_test PASSED in 14.7s //tensorflow/python/saved_model:metrics_test PASSED in 13.0s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 11.9s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 9.2s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 9.9s //tensorflow/python/saved_model:revived_types_test PASSED in 10.4s //tensorflow/python/saved_model:save_context_test PASSED in 10.0s //tensorflow/python/saved_model:save_test PASSED in 30.5s //tensorflow/python/saved_model:saved_model_test PASSED in 26.2s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 10.8s //tensorflow/python/saved_model:simple_save_test PASSED in 14.5s //tensorflow/python/saved_model:tracing_utils_test PASSED in 13.2s //tensorflow/python/saved_model:utils_test PASSED in 9.9s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 10.0s //tensorflow/python/saved_model/model_utils:export_test PASSED in 12.7s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 10.2s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 19.5s //tensorflow/python/saved_model/registration:registration_test PASSED in 10.5s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 27.2s //tensorflow/python/saved_model/tests:variable_wrapper_test PASSED in 11.7s //tensorflow/python/summary:plugin_asset_test PASSED in 9.9s //tensorflow/python/summary:summary_iterator_test PASSED in 10.3s //tensorflow/python/summary:summary_test PASSED in 11.6s //tensorflow/python/summary:summary_v2_test PASSED in 12.6s //tensorflow/python/summary/writer:writer_test PASSED in 20.9s //tensorflow/python/tools:aot_compiled_test PASSED in 24.9s //tensorflow/python/tools:freeze_graph_test PASSED in 13.8s //tensorflow/python/tools:optimize_for_inference_test PASSED in 10.3s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 28.9s //tensorflow/python/tools:saved_model_cli_test PASSED in 52.1s //tensorflow/python/tools:saved_model_utils_test PASSED in 10.3s //tensorflow/python/tools:strip_unused_test PASSED in 15.7s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 11.9s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 36.4s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 10.8s //tensorflow/python/tools/api/generator2/extractor:extractor_test PASSED in 0.7s //tensorflow/python/tools/api/generator2/generator:generator_test PASSED in 2.8s //tensorflow/python/tools/api/generator2/shared:exported_api_test PASSED in 9.8s //tensorflow/python/tpu:bfloat16_test PASSED in 9.9s //tensorflow/python/tpu:feature_column_test PASSED in 16.5s //tensorflow/python/tpu:topology_test PASSED in 10.5s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 14.0s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 12.6s //tensorflow/python/tpu:tpu_embedding_v3_utils_test PASSED in 11.1s //tensorflow/python/tpu:tpu_infeed_test PASSED in 20.6s //tensorflow/python/tpu:tpu_sharding_test PASSED in 26.7s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 12.9s //tensorflow/python/tpu/client:client_py_test PASSED in 19.9s //tensorflow/python/trackable:autotrackable_test PASSED in 13.8s //tensorflow/python/trackable:base_delegate_test PASSED in 13.5s //tensorflow/python/trackable:base_test PASSED in 10.5s //tensorflow/python/trackable:python_state_test PASSED in 11.7s //tensorflow/python/trackable:resource_test PASSED in 9.0s //tensorflow/python/trackable:trackable_utils_test PASSED in 9.2s //tensorflow/python/training:adadelta_test_cpu PASSED in 35.5s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 11.8s //tensorflow/python/training:adagrad_test_cpu PASSED in 14.6s //tensorflow/python/training:adam_test_cpu PASSED in 18.8s //tensorflow/python/training:basic_loops_test_cpu PASSED in 11.7s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 22.7s //tensorflow/python/training:checkpoint_ops_test PASSED in 9.3s //tensorflow/python/training:coordinator_test_cpu PASSED in 16.4s //tensorflow/python/training:device_setter_test_cpu PASSED in 10.1s //tensorflow/python/training:ftrl_test_cpu PASSED in 21.2s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 13.7s //tensorflow/python/training:input_test PASSED in 30.0s //tensorflow/python/training:momentum_test_cpu PASSED in 14.5s //tensorflow/python/training:monitored_session_test PASSED in 31.6s //tensorflow/python/training:moving_averages_test_cpu PASSED in 17.7s //tensorflow/python/training:optimizer_test_cpu PASSED in 14.3s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 20.5s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 14.3s //tensorflow/python/training:quantize_training_test_cpu PASSED in 28.4s //tensorflow/python/training:queue_runner_test_cpu PASSED in 10.9s //tensorflow/python/training:rmsprop_test_cpu PASSED in 45.4s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 16.0s //tensorflow/python/training:saver_test_2gpu PASSED in 56.2s //tensorflow/python/training:saver_test_cpu PASSED in 58.5s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 9.7s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 18.5s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 9.7s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 9.7s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 9.9s //tensorflow/python/training:server_lib_test PASSED in 28.3s //tensorflow/python/training:session_manager_test_cpu PASSED in 100.3s //tensorflow/python/training:slot_creator_test_cpu PASSED in 11.3s //tensorflow/python/training:supervisor_test PASSED in 17.9s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 13.7s //tensorflow/python/training:training_ops_test_cpu PASSED in 26.5s //tensorflow/python/training:training_util_test PASSED in 11.9s //tensorflow/python/training:warm_starting_util_test PASSED in 45.6s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 39.7s //tensorflow/python/training/experimental:loss_scale_test PASSED in 27.8s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 12.6s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 14.5s //tensorflow/python/util:compat_test PASSED in 26.8s //tensorflow/python/util:decorator_utils_test PASSED in 18.5s //tensorflow/python/util:deprecation_test PASSED in 9.0s //tensorflow/python/util:dispatch_test PASSED in 11.6s //tensorflow/python/util:example_parser_configuration_test PASSED in 10.7s //tensorflow/python/util:fast_module_type_test PASSED in 9.5s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 9.6s //tensorflow/python/util:function_utils_test PASSED in 14.3s //tensorflow/python/util:keyword_args_test PASSED in 11.5s //tensorflow/python/util:lazy_loader_test PASSED in 10.4s //tensorflow/python/util:lock_util_test PASSED in 12.7s //tensorflow/python/util:module_wrapper_test PASSED in 14.2s //tensorflow/python/util:nest_test PASSED in 27.7s //tensorflow/python/util:object_identity_test PASSED in 9.9s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 4.0s //tensorflow/python/util:serialization_test PASSED in 10.3s //tensorflow/python/util:tf_contextlib_test PASSED in 9.6s //tensorflow/python/util:tf_decorator_test PASSED in 10.6s //tensorflow/python/util:tf_export_test PASSED in 9.9s //tensorflow/python/util:tf_inspect_test PASSED in 11.5s //tensorflow/python/util:tf_should_use_test PASSED in 14.0s //tensorflow/python/util:tf_stack_test PASSED in 19.6s //tensorflow/python/util:traceback_utils_test PASSED in 8.9s //tensorflow/python/util:type_annotations_test PASSED in 14.0s //tensorflow/python/util:variable_utils_test PASSED in 9.3s //tensorflow/python/util:vlog_test PASSED in 10.6s //tensorflow/python/util/protobuf:protobuf_compare_test PASSED in 4.1s //tensorflow/tools/api/tests:module_test PASSED in 35.5s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.1s //tensorflow/tools/common:public_api_test PASSED in 2.9s //tensorflow/tools/common:traverse_test PASSED in 2.8s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 10.9s //tensorflow/tools/compatibility:ast_edits_test PASSED in 10.1s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 32.7s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 23.8s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 9.0s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 9.7s //tensorflow/tools/docs:tf_doctest_test PASSED in 1.9s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.7s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 1.9s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 1.6s //tensorflow/tools/graph_transforms:transforms_test PASSED in 2.5s //tensorflow/tools/proto_splitter:merge_test PASSED in 0.3s //tensorflow/tools/proto_splitter:split_graph_def_test PASSED in 9.4s //tensorflow/tools/proto_splitter:split_test PASSED in 10.6s //tensorflow/tools/proto_splitter:util_test PASSED in 10.4s //tensorflow/tools/proto_splitter/cc:composable_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:graph_def_splitter_test PASSED in 1.4s //tensorflow/tools/proto_splitter/cc:saved_model_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:util_test PASSED in 2.7s //tensorflow/tools/proto_splitter/python:saved_model_test PASSED in 10.2s //tensorflow/tools/proto_splitter/python:test_util_test PASSED in 10.0s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.1s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.9s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 10.6s Stats over 2 runs: max = 10.6s, min = 9.7s, avg = 10.2s, dev = 0.5s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 10.7s Stats over 2 runs: max = 10.7s, min = 9.7s, avg = 10.2s, dev = 0.5s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 36.7s Stats over 2 runs: max = 36.7s, min = 30.3s, avg = 33.5s, dev = 3.2s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 17.6s Stats over 2 runs: max = 17.6s, min = 16.6s, avg = 17.1s, dev = 0.5s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 32.4s Stats over 2 runs: max = 32.4s, min = 32.2s, avg = 32.3s, dev = 0.1s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 64.4s Stats over 2 runs: max = 64.4s, min = 63.3s, avg = 63.9s, dev = 0.6s //tensorflow/python/data/kernel_tests:window_test PASSED in 45.9s Stats over 2 runs: max = 45.9s, min = 33.8s, avg = 39.8s, dev = 6.1s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 28.8s Stats over 2 runs: max = 28.8s, min = 28.0s, avg = 28.4s, dev = 0.4s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 30.8s Stats over 2 runs: max = 30.8s, min = 27.2s, avg = 29.0s, dev = 1.8s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 13.0s Stats over 2 runs: max = 13.0s, min = 11.9s, avg = 12.5s, dev = 0.5s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 32.4s Stats over 2 runs: max = 32.4s, min = 20.3s, avg = 26.4s, dev = 6.1s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 154.5s Stats over 2 runs: max = 154.5s, min = 79.6s, avg = 117.0s, dev = 37.5s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 9.5s Stats over 2 runs: max = 9.5s, min = 9.4s, avg = 9.4s, dev = 0.0s //tensorflow/python/ops:control_flow_ops_test_cpu PASSED in 27.7s Stats over 2 runs: max = 27.7s, min = 23.2s, avg = 25.5s, dev = 2.2s //tensorflow/core/kernels:batch_kernels_test FLAKY, failed in 1 out of 2 in 55.2s Stats over 2 runs: max = 55.2s, min = 43.5s, avg = 49.3s, dev = 5.8s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/core/kernels/batch_kernels_test/test_attempts/attempt_1.log //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 33.2s Stats over 3 runs: max = 33.2s, min = 33.0s, avg = 33.1s, dev = 0.1s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 14.0s Stats over 3 runs: max = 14.0s, min = 13.3s, avg = 13.7s, dev = 0.3s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.3s, dev = 0.0s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 19.1s Stats over 3 runs: max = 19.1s, min = 14.7s, avg = 17.3s, dev = 1.9s //tensorflow/python/data/kernel_tests:unique_test PASSED in 20.1s Stats over 3 runs: max = 20.1s, min = 15.0s, avg = 17.2s, dev = 2.1s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 24.0s Stats over 3 runs: max = 24.0s, min = 18.7s, avg = 21.7s, dev = 2.2s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 51.3s Stats over 3 runs: max = 51.3s, min = 45.8s, avg = 47.6s, dev = 2.6s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 11.6s Stats over 3 runs: max = 11.6s, min = 11.4s, avg = 11.4s, dev = 0.1s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 16.9s Stats over 3 runs: max = 16.9s, min = 15.2s, avg = 16.1s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 369.8s Stats over 3 runs: max = 369.8s, min = 11.8s, avg = 131.3s, dev = 168.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 13.1s Stats over 3 runs: max = 13.1s, min = 12.0s, avg = 12.5s, dev = 0.5s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 20.5s Stats over 3 runs: max = 20.5s, min = 17.8s, avg = 18.9s, dev = 1.2s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 0.5s Stats over 4 runs: max = 0.5s, min = 0.5s, avg = 0.5s, dev = 0.0s //tensorflow/dtensor/python/tests:batchparallel_spmd_test_cpu PASSED in 28.1s Stats over 4 runs: max = 28.1s, min = 25.6s, avg = 26.6s, dev = 0.9s //tensorflow/dtensor/python/tests:conv_test_cpu PASSED in 64.2s Stats over 4 runs: max = 64.2s, min = 62.5s, avg = 63.3s, dev = 0.7s //tensorflow/dtensor/python/tests:sparse_test_cpu PASSED in 23.9s Stats over 4 runs: max = 23.9s, min = 16.0s, avg = 19.8s, dev = 2.8s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 54.0s Stats over 4 runs: max = 54.0s, min = 30.9s, avg = 44.7s, dev = 8.6s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 96.2s Stats over 4 runs: max = 96.2s, min = 75.8s, avg = 81.5s, dev = 8.5s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 29.5s Stats over 4 runs: max = 29.5s, min = 18.0s, avg = 23.6s, dev = 5.2s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 39.9s Stats over 4 runs: max = 39.9s, min = 24.7s, avg = 31.3s, dev = 6.0s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 75.7s Stats over 4 runs: max = 75.7s, min = 66.2s, avg = 70.4s, dev = 3.7s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 23.1s Stats over 4 runs: max = 23.1s, min = 21.9s, avg = 22.4s, dev = 0.5s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 57.0s Stats over 4 runs: max = 57.0s, min = 47.0s, avg = 51.4s, dev = 4.1s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 17.1s Stats over 4 runs: max = 17.1s, min = 11.8s, avg = 14.5s, dev = 2.6s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 33.6s Stats over 4 runs: max = 33.6s, min = 21.7s, avg = 28.8s, dev = 4.8s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 17.5s Stats over 4 runs: max = 17.5s, min = 14.4s, avg = 15.8s, dev = 1.1s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 44.2s Stats over 4 runs: max = 44.2s, min = 41.1s, avg = 42.8s, dev = 1.1s //tensorflow/python/data/kernel_tests:skip_test PASSED in 22.5s Stats over 4 runs: max = 22.5s, min = 18.4s, avg = 20.6s, dev = 1.9s //tensorflow/python/data/kernel_tests:take_test PASSED in 29.6s Stats over 4 runs: max = 29.6s, min = 28.2s, avg = 28.9s, dev = 0.5s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 46.8s Stats over 4 runs: max = 46.8s, min = 43.1s, avg = 44.5s, dev = 1.4s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 30.5s Stats over 4 runs: max = 30.5s, min = 24.4s, avg = 28.1s, dev = 2.5s //tensorflow/python/data/kernel_tests:zip_test PASSED in 20.6s Stats over 4 runs: max = 20.6s, min = 19.1s, avg = 19.6s, dev = 0.6s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 16.6s Stats over 4 runs: max = 16.6s, min = 16.0s, avg = 16.4s, dev = 0.2s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 37.3s Stats over 4 runs: max = 37.3s, min = 26.4s, avg = 30.4s, dev = 4.3s //tensorflow/python/framework:convert_to_constants_test PASSED in 35.0s Stats over 4 runs: max = 35.0s, min = 25.6s, avg = 28.5s, dev = 3.8s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 36.6s Stats over 4 runs: max = 36.6s, min = 35.0s, avg = 36.1s, dev = 0.7s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 19.2s Stats over 4 runs: max = 19.2s, min = 12.5s, avg = 15.2s, dev = 2.5s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 84.7s Stats over 4 runs: max = 84.7s, min = 40.0s, avg = 56.6s, dev = 17.4s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 37.8s Stats over 4 runs: max = 37.8s, min = 12.2s, avg = 23.7s, dev = 11.6s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 85.9s Stats over 4 runs: max = 85.9s, min = 16.1s, avg = 44.3s, dev = 27.5s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 77.4s Stats over 4 runs: max = 77.4s, min = 74.9s, avg = 76.0s, dev = 1.0s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 48.2s Stats over 4 runs: max = 48.2s, min = 36.2s, avg = 40.8s, dev = 4.9s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 103.7s Stats over 4 runs: max = 103.7s, min = 10.4s, avg = 51.1s, dev = 40.1s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 31.6s Stats over 4 runs: max = 31.6s, min = 22.8s, avg = 26.4s, dev = 3.3s //tensorflow/python/ops:nn_batchnorm_test_cpu PASSED in 19.9s Stats over 4 runs: max = 19.9s, min = 16.4s, avg = 17.7s, dev = 1.4s //tensorflow/python/ops:nn_fused_batchnorm_d9m_test_cpu PASSED in 16.8s Stats over 4 runs: max = 16.8s, min = 15.8s, avg = 16.2s, dev = 0.4s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 81.1s Stats over 4 runs: max = 81.1s, min = 20.7s, avg = 55.0s, dev = 22.0s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 48.2s Stats over 4 runs: max = 48.2s, min = 41.2s, avg = 44.4s, dev = 2.6s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 26.7s Stats over 5 runs: max = 26.7s, min = 17.8s, avg = 21.7s, dev = 3.2s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 19.1s Stats over 5 runs: max = 19.1s, min = 13.1s, avg = 15.6s, dev = 2.3s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 29.0s Stats over 5 runs: max = 29.0s, min = 25.1s, avg = 26.7s, dev = 1.7s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 22.7s Stats over 5 runs: max = 22.7s, min = 14.9s, avg = 18.1s, dev = 3.2s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 12.9s Stats over 5 runs: max = 12.9s, min = 10.2s, avg = 11.8s, dev = 1.1s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 10.9s Stats over 5 runs: max = 10.9s, min = 9.6s, avg = 10.4s, dev = 0.4s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 12.6s Stats over 5 runs: max = 12.6s, min = 11.8s, avg = 12.2s, dev = 0.3s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 16.7s Stats over 5 runs: max = 16.7s, min = 13.2s, avg = 15.0s, dev = 1.2s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 100.4s Stats over 5 runs: max = 100.4s, min = 20.1s, avg = 48.1s, dev = 27.9s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 109.4s Stats over 5 runs: max = 109.4s, min = 32.4s, avg = 52.5s, dev = 29.1s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 5.3s Stats over 5 runs: max = 5.3s, min = 3.0s, avg = 4.1s, dev = 0.9s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 28.4s Stats over 5 runs: max = 28.4s, min = 21.7s, avg = 24.6s, dev = 2.2s //tensorflow/dtensor/python/tests:multi_mesh_test_cpu PASSED in 10.9s Stats over 5 runs: max = 10.9s, min = 9.8s, avg = 10.4s, dev = 0.5s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 20.3s Stats over 5 runs: max = 20.3s, min = 14.7s, avg = 17.0s, dev = 2.1s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 17.7s Stats over 5 runs: max = 17.7s, min = 14.7s, avg = 16.5s, dev = 1.2s //tensorflow/python/distribute:vars_test_2gpu PASSED in 39.1s Stats over 5 runs: max = 39.1s, min = 37.0s, avg = 37.9s, dev = 0.8s //tensorflow/python/distribute:vars_test_cpu PASSED in 22.8s Stats over 5 runs: max = 22.8s, min = 17.2s, avg = 20.7s, dev = 1.9s //tensorflow/python/eager:device_placement_test_cpu PASSED in 26.8s Stats over 5 runs: max = 26.8s, min = 24.7s, avg = 25.8s, dev = 0.8s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 122.2s Stats over 5 runs: max = 122.2s, min = 34.6s, avg = 60.2s, dev = 32.8s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 45.7s Stats over 5 runs: max = 45.7s, min = 37.6s, avg = 41.0s, dev = 3.4s //tensorflow/python/grappler:cluster_test_cpu PASSED in 10.1s Stats over 5 runs: max = 10.1s, min = 9.4s, avg = 9.9s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 79.3s Stats over 5 runs: max = 79.3s, min = 57.3s, avg = 68.7s, dev = 7.5s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 75.3s Stats over 5 runs: max = 75.3s, min = 58.5s, avg = 65.8s, dev = 5.5s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 100.0s Stats over 5 runs: max = 100.0s, min = 94.1s, avg = 96.3s, dev = 2.2s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 50.4s Stats over 5 runs: max = 50.4s, min = 47.8s, avg = 49.0s, dev = 1.0s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 57.8s Stats over 5 runs: max = 57.8s, min = 56.7s, avg = 57.0s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 71.4s Stats over 5 runs: max = 71.4s, min = 60.2s, avg = 66.0s, dev = 4.3s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 62.4s Stats over 5 runs: max = 62.4s, min = 60.6s, avg = 61.5s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 61.2s Stats over 5 runs: max = 61.2s, min = 57.7s, avg = 58.9s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 41.8s Stats over 5 runs: max = 41.8s, min = 36.9s, avg = 40.0s, dev = 1.7s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 72.8s Stats over 5 runs: max = 72.8s, min = 67.9s, avg = 69.8s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 51.7s Stats over 5 runs: max = 51.7s, min = 50.8s, avg = 51.3s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 69.2s Stats over 5 runs: max = 69.2s, min = 67.0s, avg = 68.3s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 147.5s Stats over 5 runs: max = 147.5s, min = 11.1s, avg = 38.8s, dev = 54.4s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 20.5s Stats over 5 runs: max = 20.5s, min = 12.0s, avg = 14.8s, dev = 3.1s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 17.8s Stats over 5 runs: max = 17.8s, min = 11.2s, avg = 12.8s, dev = 2.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 43.5s Stats over 5 runs: max = 43.5s, min = 12.5s, avg = 19.3s, dev = 12.1s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 73.6s Stats over 5 runs: max = 73.6s, min = 34.6s, avg = 52.8s, dev = 13.1s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 35.4s Stats over 6 runs: max = 35.4s, min = 23.2s, avg = 29.1s, dev = 3.9s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 23.6s Stats over 6 runs: max = 23.6s, min = 16.2s, avg = 18.9s, dev = 2.6s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 29.6s Stats over 6 runs: max = 29.6s, min = 10.3s, avg = 18.5s, dev = 7.7s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 60.8s Stats over 6 runs: max = 60.8s, min = 10.4s, avg = 21.5s, dev = 17.6s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 60.4s Stats over 6 runs: max = 60.4s, min = 21.6s, avg = 44.8s, dev = 13.9s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 13.7s Stats over 7 runs: max = 13.7s, min = 9.2s, avg = 11.2s, dev = 1.7s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 20.8s Stats over 8 runs: max = 20.8s, min = 16.9s, avg = 18.8s, dev = 1.1s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 99.6s Stats over 8 runs: max = 99.6s, min = 26.8s, avg = 51.2s, dev = 25.0s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 81.4s Stats over 8 runs: max = 81.4s, min = 8.6s, avg = 36.1s, dev = 27.0s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 19.5s Stats over 8 runs: max = 19.5s, min = 11.9s, avg = 14.8s, dev = 2.6s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 34.8s Stats over 8 runs: max = 34.8s, min = 25.3s, avg = 28.6s, dev = 3.0s //tensorflow/dtensor/python/tests:input_util_test PASSED in 30.5s Stats over 8 runs: max = 30.5s, min = 17.8s, avg = 24.4s, dev = 4.0s //tensorflow/dtensor/python/tests:save_restore_v2_test_cpu PASSED in 33.3s Stats over 8 runs: max = 33.3s, min = 24.1s, avg = 26.8s, dev = 3.5s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 35.2s Stats over 8 runs: max = 35.2s, min = 15.0s, avg = 21.7s, dev = 7.1s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 41.2s Stats over 8 runs: max = 41.2s, min = 20.2s, avg = 30.4s, dev = 7.3s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 64.8s Stats over 8 runs: max = 64.8s, min = 27.8s, avg = 43.5s, dev = 14.4s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 26.2s Stats over 8 runs: max = 26.2s, min = 11.4s, avg = 18.1s, dev = 6.3s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 22.6s Stats over 8 runs: max = 22.6s, min = 8.7s, avg = 14.0s, dev = 4.9s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_load_ft_test PASSED in 32.3s Stats over 8 runs: max = 32.3s, min = 22.7s, avg = 26.1s, dev = 3.0s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_load_test PASSED in 88.4s Stats over 8 runs: max = 88.4s, min = 55.3s, avg = 73.2s, dev = 11.9s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 18.2s Stats over 8 runs: max = 18.2s, min = 10.1s, avg = 12.6s, dev = 2.6s //tensorflow/python/data/kernel_tests:batch_test PASSED in 43.6s Stats over 8 runs: max = 43.6s, min = 37.3s, avg = 40.7s, dev = 2.3s //tensorflow/python/data/kernel_tests:filter_test PASSED in 19.6s Stats over 8 runs: max = 19.6s, min = 15.3s, avg = 17.6s, dev = 1.4s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 37.8s Stats over 8 runs: max = 37.8s, min = 21.2s, avg = 28.3s, dev = 5.0s //tensorflow/python/data/kernel_tests:shard_test PASSED in 31.4s Stats over 8 runs: max = 31.4s, min = 20.5s, avg = 26.3s, dev = 3.2s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 108.8s Stats over 8 runs: max = 108.8s, min = 85.1s, avg = 89.0s, dev = 7.5s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 37.0s Stats over 8 runs: max = 37.0s, min = 21.8s, avg = 28.9s, dev = 4.2s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 97.7s Stats over 8 runs: max = 97.7s, min = 14.3s, avg = 35.9s, dev = 29.7s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 49.9s Stats over 8 runs: max = 49.9s, min = 29.6s, avg = 42.7s, dev = 6.6s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 167.7s Stats over 8 runs: max = 167.7s, min = 117.3s, avg = 150.4s, dev = 15.3s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 93.7s Stats over 8 runs: max = 93.7s, min = 72.4s, avg = 83.1s, dev = 8.5s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 69.3s Stats over 8 runs: max = 69.3s, min = 7.6s, avg = 17.8s, dev = 20.1s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 9.5s Stats over 8 runs: max = 9.5s, min = 7.0s, avg = 8.6s, dev = 0.9s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 69.1s Stats over 8 runs: max = 69.1s, min = 51.3s, avg = 59.2s, dev = 6.8s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 44.0s Stats over 8 runs: max = 44.0s, min = 31.3s, avg = 35.1s, dev = 3.8s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 10.1s Stats over 10 runs: max = 10.1s, min = 8.3s, avg = 9.3s, dev = 0.5s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 13.9s Stats over 10 runs: max = 13.9s, min = 12.1s, avg = 13.0s, dev = 0.5s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 16.2s Stats over 10 runs: max = 16.2s, min = 9.4s, avg = 12.9s, dev = 2.1s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 35.1s Stats over 10 runs: max = 35.1s, min = 14.9s, avg = 24.0s, dev = 6.1s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 106.2s Stats over 10 runs: max = 106.2s, min = 55.2s, avg = 75.5s, dev = 15.7s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 83.5s Stats over 10 runs: max = 83.5s, min = 44.5s, avg = 60.6s, dev = 15.3s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 40.3s Stats over 10 runs: max = 40.3s, min = 8.5s, avg = 19.7s, dev = 11.0s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 37.1s Stats over 10 runs: max = 37.1s, min = 25.7s, avg = 31.3s, dev = 3.9s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 28.1s Stats over 10 runs: max = 28.1s, min = 16.1s, avg = 22.2s, dev = 4.4s //tensorflow/python/framework:function_test_cpu PASSED in 68.5s Stats over 10 runs: max = 68.5s, min = 10.4s, avg = 21.8s, dev = 17.0s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 15.1s Stats over 10 runs: max = 15.1s, min = 9.7s, avg = 12.2s, dev = 2.0s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 10.0s Stats over 10 runs: max = 10.0s, min = 9.2s, avg = 9.6s, dev = 0.2s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 13.1s Stats over 10 runs: max = 13.1s, min = 5.8s, avg = 9.0s, dev = 2.8s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 114.6s Stats over 10 runs: max = 114.6s, min = 104.4s, avg = 111.5s, dev = 3.1s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 77.1s Stats over 10 runs: max = 77.1s, min = 24.2s, avg = 49.7s, dev = 17.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 9.5s Stats over 10 runs: max = 9.5s, min = 9.3s, avg = 9.4s, dev = 0.1s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 15.6s Stats over 10 runs: max = 15.6s, min = 6.2s, avg = 12.1s, dev = 2.7s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 27.4s Stats over 10 runs: max = 27.4s, min = 8.6s, avg = 17.2s, dev = 7.6s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 24.1s Stats over 10 runs: max = 24.1s, min = 10.1s, avg = 13.4s, dev = 4.8s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 27.0s Stats over 10 runs: max = 27.0s, min = 12.6s, avg = 19.9s, dev = 5.6s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 15.2s Stats over 10 runs: max = 15.2s, min = 13.5s, avg = 14.4s, dev = 0.6s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 132.6s Stats over 10 runs: max = 132.6s, min = 34.8s, avg = 83.1s, dev = 46.9s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 56.5s Stats over 10 runs: max = 56.5s, min = 12.2s, avg = 19.2s, dev = 12.6s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 11.2s Stats over 10 runs: max = 11.2s, min = 6.6s, avg = 9.6s, dev = 1.4s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 126.8s Stats over 10 runs: max = 126.8s, min = 117.3s, avg = 122.0s, dev = 2.7s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 27.4s Stats over 10 runs: max = 27.4s, min = 25.1s, avg = 26.6s, dev = 0.8s //tensorflow/python/saved_model:load_test_cpu PASSED in 56.2s Stats over 10 runs: max = 56.2s, min = 34.0s, avg = 39.3s, dev = 6.1s //tensorflow/python/distribute/failure_handling:failure_handler_test FLAKY, failed in 2 out of 10 in 72.3s Stats over 10 runs: max = 72.3s, min = 30.8s, avg = 52.2s, dev = 11.9s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_5_of_8/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_1_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:fft_test_cpu PASSED in 24.7s Stats over 12 runs: max = 24.7s, min = 13.5s, avg = 19.1s, dev = 4.1s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 14.0s Stats over 12 runs: max = 14.0s, min = 4.7s, avg = 9.5s, dev = 3.0s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 16.1s Stats over 12 runs: max = 16.1s, min = 10.0s, avg = 12.1s, dev = 2.0s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 13.8s Stats over 12 runs: max = 13.8s, min = 5.4s, avg = 10.0s, dev = 2.0s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 18.0s Stats over 12 runs: max = 18.0s, min = 11.2s, avg = 14.9s, dev = 2.5s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 21.6s Stats over 12 runs: max = 21.6s, min = 15.1s, avg = 18.6s, dev = 1.5s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 230.1s Stats over 12 runs: max = 230.1s, min = 19.3s, avg = 56.9s, dev = 58.3s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 247.4s Stats over 12 runs: max = 247.4s, min = 31.2s, avg = 70.3s, dev = 59.4s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 28.4s Stats over 15 runs: max = 28.4s, min = 14.6s, avg = 23.4s, dev = 4.1s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 151.4s Stats over 15 runs: max = 151.4s, min = 142.7s, avg = 147.1s, dev = 2.8s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 58.0s Stats over 15 runs: max = 58.0s, min = 16.4s, avg = 22.1s, dev = 10.5s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 29.0s Stats over 16 runs: max = 29.0s, min = 7.3s, avg = 20.8s, dev = 7.2s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 32.5s Stats over 16 runs: max = 32.5s, min = 15.4s, avg = 23.0s, dev = 4.6s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 34.9s Stats over 16 runs: max = 34.9s, min = 9.2s, avg = 13.4s, dev = 5.8s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 10.8s Stats over 16 runs: max = 10.8s, min = 6.4s, avg = 8.8s, dev = 1.2s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 19.9s Stats over 16 runs: max = 19.9s, min = 8.7s, avg = 12.3s, dev = 2.3s //tensorflow/python/ops:image_ops_test_cpu PASSED in 24.3s Stats over 16 runs: max = 24.3s, min = 11.9s, avg = 16.6s, dev = 3.4s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 63.5s Stats over 17 runs: max = 63.5s, min = 7.4s, avg = 25.6s, dev = 15.5s //tensorflow/python/data/kernel_tests:map_test PASSED in 50.7s Stats over 19 runs: max = 50.7s, min = 16.0s, avg = 34.2s, dev = 10.9s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 10.3s Stats over 20 runs: max = 10.3s, min = 4.1s, avg = 6.6s, dev = 2.1s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 10.4s Stats over 20 runs: max = 10.4s, min = 4.5s, avg = 6.5s, dev = 2.4s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 15.4s Stats over 20 runs: max = 15.4s, min = 4.1s, avg = 8.0s, dev = 2.6s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 25.2s Stats over 20 runs: max = 25.2s, min = 7.5s, avg = 12.6s, dev = 4.5s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 11.3s Stats over 20 runs: max = 11.3s, min = 5.3s, avg = 7.3s, dev = 1.6s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 22.9s Stats over 20 runs: max = 22.9s, min = 4.4s, avg = 9.8s, dev = 4.5s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 23.5s Stats over 20 runs: max = 23.5s, min = 7.3s, avg = 12.4s, dev = 3.9s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 21.5s Stats over 20 runs: max = 21.5s, min = 6.5s, avg = 12.5s, dev = 4.1s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 30.2s Stats over 20 runs: max = 30.2s, min = 16.6s, avg = 24.1s, dev = 3.5s //tensorflow/python/kernel_tests:metrics_test PASSED in 51.3s Stats over 20 runs: max = 51.3s, min = 18.8s, avg = 28.3s, dev = 9.6s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 9.6s Stats over 20 runs: max = 9.6s, min = 5.6s, avg = 8.5s, dev = 1.0s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 22.5s Stats over 20 runs: max = 22.5s, min = 5.3s, avg = 11.7s, dev = 6.5s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 58.1s Stats over 20 runs: max = 58.1s, min = 4.6s, avg = 19.3s, dev = 15.9s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 99.0s Stats over 20 runs: max = 99.0s, min = 27.1s, avg = 51.2s, dev = 20.2s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 11.6s Stats over 20 runs: max = 11.6s, min = 5.8s, avg = 8.7s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 15.6s Stats over 20 runs: max = 15.6s, min = 5.7s, avg = 10.4s, dev = 2.8s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 212.6s Stats over 20 runs: max = 212.6s, min = 47.7s, avg = 119.7s, dev = 54.7s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 37.1s Stats over 20 runs: max = 37.1s, min = 4.2s, avg = 18.9s, dev = 11.0s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 33.2s Stats over 20 runs: max = 33.2s, min = 5.6s, avg = 15.8s, dev = 7.5s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 32.9s Stats over 20 runs: max = 32.9s, min = 15.4s, avg = 22.7s, dev = 4.6s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 66.6s Stats over 20 runs: max = 66.6s, min = 11.5s, avg = 33.2s, dev = 19.3s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 34.9s Stats over 20 runs: max = 34.9s, min = 20.3s, avg = 26.3s, dev = 4.4s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 21.4s Stats over 24 runs: max = 21.4s, min = 8.1s, avg = 14.4s, dev = 4.6s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 21.5s Stats over 24 runs: max = 21.5s, min = 8.7s, avg = 13.6s, dev = 4.0s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 102.5s Stats over 25 runs: max = 102.5s, min = 79.9s, avg = 88.9s, dev = 5.8s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 29.5s Stats over 30 runs: max = 29.5s, min = 5.0s, avg = 19.4s, dev = 9.5s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 23.7s Stats over 32 runs: max = 23.7s, min = 4.7s, avg = 11.6s, dev = 5.0s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 41.1s Stats over 32 runs: max = 41.1s, min = 4.8s, avg = 18.2s, dev = 11.0s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.7s Stats over 48 runs: max = 1.7s, min = 0.4s, avg = 0.6s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 103.6s Stats over 50 runs: max = 103.6s, min = 29.2s, avg = 60.0s, dev = 25.6s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 21.5s Stats over 50 runs: max = 21.5s, min = 4.5s, avg = 10.9s, dev = 3.7s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 32.9s Stats over 50 runs: max = 32.9s, min = 5.2s, avg = 16.0s, dev = 7.1s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 68.1s Stats over 50 runs: max = 68.1s, min = 34.2s, avg = 42.0s, dev = 7.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 15.9s Stats over 50 runs: max = 15.9s, min = 5.2s, avg = 9.8s, dev = 2.9s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 30.6s Stats over 50 runs: max = 30.6s, min = 8.6s, avg = 16.2s, dev = 5.4s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 9.4s Stats over 50 runs: max = 9.4s, min = 3.5s, avg = 5.2s, dev = 1.5s Executed 3061 out of 3061 tests: 3061 tests pass. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.