==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 8 of 8): 2023-12-27 04:34:55.763020: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.11.6: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24885 I1227 04:35:03.116754 281473465807904 test_util.py:3999] Using local port 24885 INFO:tensorflow:Using local port 24884 I1227 04:35:03.118559 281473465807904 test_util.py:3999] Using local port 24884 INFO:tensorflow:Using local port 24883 I1227 04:35:03.120012 281473465807904 test_util.py:3999] Using local port 24883 INFO:tensorflow:Using local port 24882 I1227 04:35:03.121423 281473465807904 test_util.py:3999] Using local port 24882 2023-12-27 04:35:04.768253: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2023-12-27 04:35:05.008536: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. INFO:tensorflow:Cluster starting. I1227 04:35:07.621208 281473465807904 failure_handler_test.py:432] Cluster starting. [worker-0]: I1227 04:35:07.673327 281473202811936 multi_process_runner.py:840] Subprocess with PID 546088 (worker, 0) is now being started. [worker-1]: I1227 04:35:07.678110 281473202811936 multi_process_runner.py:840] Subprocess with PID 546115 (worker, 1) is now being started. [worker-0]: I1227 04:35:07.673723 281473202811936 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24885", "localhost:24884", "localhost:24883", "localhost:24882"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I1227 04:35:07.686947 281473202811936 multi_process_runner.py:840] Subprocess with PID 546127 (worker, 2) is now being started. [worker-3]: I1227 04:35:07.694057 281473202811936 multi_process_runner.py:840] Subprocess with PID 546145 (worker, 3) is now being started. [worker-1]: I1227 04:35:07.678502 281473202811936 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24885", "localhost:24884", "localhost:24883", "localhost:24882"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2023-12-27 04:35:07.709648: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24885 [worker-2]: I1227 04:35:07.687344 281473202811936 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24885", "localhost:24884", "localhost:24883", "localhost:24882"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: 2023-12-27 04:35:07.716594: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24884 [worker-3]: I1227 04:35:07.694446 281473202811936 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24885", "localhost:24884", "localhost:24883", "localhost:24882"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-12-27 04:35:07.718028: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12527188944681571285 [worker-2]: 2023-12-27 04:35:07.729096: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24883 [worker-0]: 2023-12-27 04:35:07.718057: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-3]: 2023-12-27 04:35:07.731304: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24882 [worker-0]: 2023-12-27 04:35:07.718068: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: /job:worker/replica:0/task:2 [worker-3]: I0000 00:00:1703651707.733210 546145 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: /job:worker/replica:0/task:1 [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: /job:worker/replica:0/task:3 [worker-3]: I1227 04:35:07.735017 281473202811936 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: I0000 00:00:1703651707.721398 546115 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-1]: I1227 04:35:07.735010 281473202811936 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0000 00:00:1703651707.718208 546088 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:07.721066: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2987647253675140145 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:07.721092: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: 2023-12-27 04:35:07.721102: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-1]: I1227 04:35:07.803364 281473202811936 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: I1227 04:35:07.803365 281473202811936 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: /job:worker/replica:0/task:2 [worker-3]: INFO:tensorflow:Check health not enabled. [worker-0]: /job:worker/replica:0/task:3 [worker-3]: I1227 04:35:07.803881 281473202811936 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1227 04:35:07.803881 281473202811936 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: I1227 04:35:07.804117 281473202811936 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: 2023-12-27 04:35:07.732981: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 6713843705263446899 [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: 2023-12-27 04:35:07.733009: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-2]: I0000 00:00:1703651707.733171 546127 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:07.733020: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1227 04:35:07.804119 281473202811936 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1227 04:35:07.735561 281473202811936 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: /job:worker/replica:0/task:3 [worker-0]: 2023-12-27 04:35:07.733070: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 10892692515577013874 [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: 2023-12-27 04:35:07.733089: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1227 04:35:07.803367 281473202811936 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: I1227 04:35:07.735137 281473202811936 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Check health not enabled. [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: I1227 04:35:07.803901 281473202811936 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: I1227 04:35:07.803365 281473202811936 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Check health not enabled. [worker-2]: I1227 04:35:07.804135 281473202811936 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1227 04:35:07.803881 281473202811936 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1227 04:35:07.804119 281473202811936 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24885', 'localhost:24884', 'localhost:24883', 'localhost:24882']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1227 04:35:07.874290 281473202811936 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I1227 04:35:07.875460 281473202811936 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I1227 04:35:07.875498 281473202811936 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1227 04:35:07.876135 281473202811936 failure_handling.py:674] Start watcher for local signal. [worker-0]: I1227 04:35:07.875053 281473202811936 failure_handling.py:674] Start watcher for local signal. [worker-1]: I1227 04:35:07.876168 281473202811936 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I1227 04:35:07.876379 281473202811936 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1227 04:35:07.875303 281473202811936 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: I1227 04:35:07.876408 281473202811936 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-1]: Instructions for updating: [worker-3]: I1227 04:35:07.880671 281473202811936 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1227 04:35:07.876757 281473202811936 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-2]: W1227 04:35:07.876706 281473202811936 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-3]: I1227 04:35:07.881635 281473202811936 failure_handling.py:674] Start watcher for local signal. [worker-0]: W1227 04:35:07.875629 281473202811936 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: Instructions for updating: [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: I1227 04:35:07.881917 281473202811936 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:Start training at 0 [worker-2]: INFO:tensorflow:Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I1227 04:35:07.876890 281473202811936 failure_handler_test.py:197] Start training at 0 [worker-1]: I1227 04:35:07.876942 281473202811936 failure_handler_test.py:197] Start training at 0 [worker-0]: I1227 04:35:07.875815 281473202811936 failure_handler_test.py:197] Start training at 0 [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1227 04:35:07.882319 281473202811936 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I1227 04:35:07.882516 281473202811936 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:20.297259 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:20.309416 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:20.313927 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:20.306187 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:20.406538 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:20.406511 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:20.406819 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:20.406806 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:20.466953 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:20.466405 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:20.466406 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:20.498024 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:20.809145 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:20.809257 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:20.809145 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:20.812246 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:20.871258 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:20.871361 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:20.871293 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:20.871898 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff781553a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff781553a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:20.922244 281473202811936 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff781553a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1227 04:35:20.922435 281473202811936 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff781553a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff781559e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1227 04:35:20.922732 281473202811936 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff781559e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff78155ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1227 04:35:20.922587 281473202811936 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff78155ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:20.931195 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:20.931276 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:20.931325 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:20.935468 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff78155ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff78155f80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1227 04:35:20.996976 281473202811936 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff78155ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1227 04:35:20.997176 281473202811936 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff78155f80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff78154180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1227 04:35:21.001716 281473202811936 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff78154180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.005831 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.005937 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.010358 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff78155ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:21.015345 281473202811936 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff78155ee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.025460 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.089034 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.089030 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.089253 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.097730 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.178511 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.194267 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.194095 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.194333 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.254450 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.254526 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.270554 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.290133 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.387228 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.402216 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.419529 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.419903 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.491399 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.503951 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.512508 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.519334 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.640084 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.664671 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.668110 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.746614 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.915405 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.918900 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.915439 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.934710 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.026271 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.024910 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.025476 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.059250 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-0]: I1227 04:35:22.119989 281473202811936 failure_handler_test.py:195] epoch 0 finished [worker-1]: I1227 04:35:22.120199 281473202811936 failure_handler_test.py:195] epoch 0 finished [worker-2]: I1227 04:35:22.120918 281473202811936 failure_handler_test.py:195] epoch 0 finished [worker-3]: I1227 04:35:22.119905 281473202811936 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.128962 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.131118 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.131073 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.129650 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.192612 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.196027 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.193200 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.209471 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.292297 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.303860 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.320196 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.321380 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.382825 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.382869 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.398129 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.400299 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.474002 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.475477 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.475501 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.474233 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.555701 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.565992 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.569498 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.579230 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.698238 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.698306 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.725552 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.759232 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.817293 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.820830 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.833720 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.836051 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.904297 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.904057 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.954004 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.938867 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.037863 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.042671 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.038607 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.033288 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.114203 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.118291 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.116260 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.116243 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending SIGTERM [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1227 04:35:23.699990 281473465807904 failure_handler_test.py:438] sending SIGTERM [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_MWMSmultiworker): 31.85s [worker-1]: I1227 04:35:23.196988 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1227 04:35:34.948502 281473465807904 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_MWMSmultiworker): 31.85s [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSlocal [worker-0]: I1227 04:35:23.197023 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.197731 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.217591 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.277823 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.279151 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.279373 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.278307 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.337772 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.337707 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.337708 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.394421 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I1227 04:35:23.337591 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.395210 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.443150 281473202811936 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: I1227 04:35:23.395310 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.394201 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-0]: I1227 04:35:23.443029 281473202811936 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.443121 281473202811936 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.442749 281473202811936 failure_handler_test.py:195] epoch 1 finished [worker-2]: I1227 04:35:23.451616 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.452424 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.450927 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.508395 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.451961 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.508210 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.507464 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.506697 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.563396 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.563456 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.563430 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.563974 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.623214 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.623208 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.623242 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.623653 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.680041 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.680036 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.679729 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.680076 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.735499 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.734585 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.735992 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.735575 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.790913 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.790026 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.790603 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.791375 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.844261 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.845102 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.845321 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.899491 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.845216 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.899668 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.898433 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.899512 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.954000 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.952767 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.953809 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.953861 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.007105 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.008385 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.008102 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.008185 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.060980 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.062201 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.062019 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.115596 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.116840 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.062052 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.116837 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.172140 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.117788 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.172760 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.172650 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.227548 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.226593 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-1]: I1227 04:35:24.173517 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.227404 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: I1227 04:35:24.275230 281473202811936 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I1227 04:35:24.274784 281473202811936 failure_handler_test.py:195] epoch 2 finished [worker-1]: I1227 04:35:24.227000 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.275033 281473202811936 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.282402 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.283536 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.275128 281473202811936 failure_handler_test.py:195] epoch 2 finished [worker-0]: I1227 04:35:24.283291 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.337608 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.336609 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.282943 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.337413 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.391037 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.337035 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.390594 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.390903 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.445361 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.445131 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.390812 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.499090 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.444855 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I1227 04:35:35.038987 281473465807904 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I1227 04:35:35.039571 281473465807904 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.498927 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.445138 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.498267 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.553068 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.552933 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.498606 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.552279 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.606470 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.553072 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.605823 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.659876 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.605595 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.659177 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Start watcher for local signal. I1227 04:35:35.066586 281473465807904 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1227 04:35:24.606513 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1227 04:35:35.066898 281473465807904 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W1227 04:35:35.067610 281473465807904 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I1227 04:35:24.713577 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Start training at 0 I1227 04:35:35.067907 281473465807904 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.659722 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.659858 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.712172 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.766648 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.712924 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.712516 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.766096 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.819433 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.766585 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.818916 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.766437 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.871932 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.819202 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.871464 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.819253 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.926664 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.871815 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.926180 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.871843 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.979136 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.926312 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.926531 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.978737 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.032552 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.978944 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: I1227 04:35:24.979047 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.079020 281473202811936 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.032270 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.032652 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.033005 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.086959 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.078660 281473202811936 failure_handler_test.py:195] epoch 3 finished [worker-1]: I1227 04:35:25.078967 281473202811936 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.140050 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.078831 281473202811936 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.086582 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.086618 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.086930 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.193794 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.139655 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.139977 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.139848 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.247600 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.193854 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.193653 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.194107 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.301097 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.247230 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.247429 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.354441 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.247560 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.300677 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.300888 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.407406 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.300956 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.353946 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.354214 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.460159 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.354311 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.406975 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.407242 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.512375 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.407219 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.459531 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.459959 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.567097 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.459912 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.511722 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.512150 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.619237 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.512090 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.566493 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.671785 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.566928 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.566783 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.723778 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.618697 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.619121 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.618963 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.776368 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.671153 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.671596 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.828328 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.671452 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.723222 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.723610 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.873816 281473202811936 failure_handler_test.py:195] epoch 4 finished [worker-1]: I1227 04:35:25.723476 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.775795 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.776233 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.881525 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.776117 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.827896 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.828292 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-2]: I1227 04:35:25.933454 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.828063 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: I1227 04:35:25.873444 281473202811936 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: I1227 04:35:25.873639 281473202811936 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.986492 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.873732 281473202811936 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.880734 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.881230 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.038625 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.881203 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.933277 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.933694 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.933207 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.091265 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.985558 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.986342 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.985926 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.143380 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.037980 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.038850 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.090732 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.038201 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.197639 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.091138 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.142653 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.250058 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.091034 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.143387 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.301982 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.196989 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.142954 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.197465 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.354402 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.248970 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.197725 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.406325 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.249463 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.301493 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.458196 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.301843 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.249255 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.353476 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.510091 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.353899 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.302089 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.405781 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.562022 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.406149 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.353698 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.457576 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.613860 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.458047 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.406068 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-3]: I1227 04:35:26.509474 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.658739 281473202811936 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.509907 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.457920 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.561422 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.666522 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.561870 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.509817 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.613272 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-2]: I1227 04:35:26.718443 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.561715 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.613730 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.658346 281473202811936 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.770992 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.658544 281473202811936 failure_handler_test.py:195] epoch 5 finished [worker-1]: I1227 04:35:26.613568 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.665642 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-2]: I1227 04:35:26.824934 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.666182 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.658650 281473202811936 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.717920 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.876782 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.718293 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.666047 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.770192 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.928583 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.770761 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.718210 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.824447 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.980462 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.824766 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.770442 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.876358 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.032340 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.876572 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.824783 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.928160 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.928489 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.084030 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.876642 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.979921 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.980221 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.135891 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.928437 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.031926 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.032232 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.187735 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.980237 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.083579 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.083854 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.239820 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.032265 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.135461 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.135765 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.291647 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.083902 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.187494 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.187179 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.343431 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.135785 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.239660 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.239351 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.395813 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.187568 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: I1227 04:35:27.291421 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.442736 281473202811936 failure_handler_test.py:195] epoch 6 finished [worker-3]: I1227 04:35:27.291205 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.239653 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.343272 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.450338 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.291491 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.342955 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.502979 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.395596 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-1]: I1227 04:35:27.343302 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.554881 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.395339 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.442549 281473202811936 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.395717 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-2]: I1227 04:35:27.606846 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.450086 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-3]: I1227 04:35:27.442394 281473202811936 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.442682 281473202811936 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.658775 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.502779 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.450160 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.450546 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.710916 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.554660 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.502653 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.502954 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.762706 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.606675 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.554534 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.554831 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.814663 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.606413 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.658620 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.866559 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.606719 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.658375 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.710674 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.918416 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.658719 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.710434 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.970415 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.762521 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.710774 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.762234 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.022522 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.814416 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.814152 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.762603 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.076541 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.866409 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.866047 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.128457 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.918202 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.917855 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.180626 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.814499 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.969993 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.225520 281473202811936 failure_handler_test.py:195] epoch 7 finished [worker-0]: I1227 04:35:27.970289 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.866431 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.022129 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.226627 281473202811936 failure_handler_test.py:245] Training finished. [worker-0]: I1227 04:35:28.022279 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.918178 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.076215 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.076402 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.970331 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.127946 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.128278 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.022507 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.180631 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-0]: I1227 04:35:28.180456 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.076461 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.225132 281473202811936 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Training finished. [worker-0]: I1227 04:35:28.225340 281473202811936 failure_handler_test.py:195] epoch 7 finished [worker-1]: I1227 04:35:28.128260 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.225764 281473202811936 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.226733 281473202811936 failure_handler_test.py:245] Training finished. [worker-1]: I1227 04:35:28.180499 281473202811936 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I1227 04:35:28.225452 281473202811936 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I1227 04:35:28.226385 281473202811936 failure_handler_test.py:245] Training finished. WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef85df2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1227 04:35:35.606722 281473465807904 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef85df2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef85dff60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1227 04:35:35.637979 281473465807904 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef85dff60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I1227 04:35:35.941243 281473465807904 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I1227 04:35:36.394072 281473465807904 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I1227 04:35:36.823395 281473465807904 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I1227 04:35:37.258387 281473465807904 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I1227 04:35:37.624804 281473465807904 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I1227 04:35:37.986917 281473465807904 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I1227 04:35:38.354513 281473465807904 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I1227 04:35:38.747880 281473465807904 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I1227 04:35:38.748318 281473465807904 failure_handler_test.py:245] Training finished. INFO:tensorflow:sending sigterm I1227 04:35:38.966744 281470571573728 failure_handler_test.py:327] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I1227 04:35:38.975120 281473465807904 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSlocal): 4.02s I1227 04:35:38.975904 281473465807904 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSlocal): 4.02s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSlocal [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSlocal INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I1227 04:35:39.027506 281473465807904 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I1227 04:35:39.027961 281473465807904 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start watcher for local signal. I1227 04:35:39.043389 281473465807904 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I1227 04:35:39.043909 281473465807904 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. INFO:tensorflow:Start training at 0 I1227 04:35:39.044161 281473465807904 failure_handler_test.py:197] Start training at 0 INFO:tensorflow:epoch 0 finished I1227 04:35:39.622267 281473465807904 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I1227 04:35:40.254675 281473465807904 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I1227 04:35:40.634150 281473465807904 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I1227 04:35:41.039331 281473465807904 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I1227 04:35:41.432212 281473465807904 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I1227 04:35:41.984739 281473465807904 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:sending sigterm I1227 04:35:42.015216 281470571573728 failure_handler_test.py:327] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I1227 04:35:42.017333 281473465807904 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:Termination caught in main thread on preempted worker I1227 04:35:42.034690 281473465807904 failure_handling.py:1159] Termination caught in main thread on preempted worker INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. I1227 04:35:42.059251 281473465807904 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/701db649fbdbb7db03a619b1196667f9kw2tful8/tmpo1x0knj_/fh_ckpt I1227 04:35:42.124677 281473465807904 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/701db649fbdbb7db03a619b1196667f9kw2tful8/tmpo1x0knj_/fh_ckpt INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. I1227 04:35:42.125006 281473465807904 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I1227 04:35:42.125322 281473465807904 failure_handler_test.py:351] restarting workers INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I1227 04:35:42.128147 281473465807904 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I1227 04:35:42.128423 281473465807904 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start watcher for local signal. I1227 04:35:42.153354 281473465807904 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I1227 04:35:42.153818 281473465807904 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. INFO:tensorflow:Start training at 93 I1227 04:35:42.154055 281473465807904 failure_handler_test.py:197] Start training at 93 INFO:tensorflow:training restarted I1227 04:35:42.154232 281473465807904 failure_handler_test.py:207] training restarted INFO:tensorflow:epoch 6 finished I1227 04:35:43.249790 281473465807904 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I1227 04:35:44.071518 281473465807904 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I1227 04:35:44.071862 281473465807904 failure_handler_test.py:245] Training finished. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSlocal): 5.09s I1227 04:35:44.073451 281473465807904 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSlocal): 5.09s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSlocal ====================================================================== ERROR: test_grace_period_continue_training_test_inputarg_manager_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_MWMSmultiworker test_grace_period_continue_training_test_inputarg_manager_strategyoption_MWMSmultiworker(input_arg='manager', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 439, in test_grace_period_continue_training os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process ---------------------------------------------------------------------- Ran 3 tests in 40.974s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 5 of 8): 2023-12-27 04:34:55.614241: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.11.6: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I1227 04:35:03.188864 281473083339808 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I1227 04:35:03.189246 281473083339808 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W1227 04:35:03.189545 281473083339808 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I1227 04:35:03.189769 281473083339808 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee31d1440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1227 04:35:03.487402 281473083339808 polymorphic_function.py:157] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee31d1440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee31d1440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1227 04:35:03.500936 281473083339808 polymorphic_function.py:157] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee31d1440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I1227 04:35:03.614213 281473083339808 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I1227 04:35:03.811736 281473083339808 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I1227 04:35:04.044735 281473083339808 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I1227 04:35:04.483119 281473083339808 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I1227 04:35:04.672344 281473083339808 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I1227 04:35:04.881002 281473083339808 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I1227 04:35:05.069409 281473083339808 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:sending sigterm I1227 04:35:05.105038 281470197035488 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I1227 04:35:05.105504 281473083339808 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:Termination caught in main thread on preempted worker I1227 04:35:05.109891 281473083339808 failure_handling.py:1159] Termination caught in main thread on preempted worker INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. I1227 04:35:05.122607 281473083339808 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpu916prp8/fh_ckpt I1227 04:35:05.183920 281473083339808 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpu916prp8/fh_ckpt INFO:tensorflow:Continue training for the grace period. I1227 04:35:05.184208 281473083339808 failure_handling.py:1134] Continue training for the grace period. INFO:tensorflow:epoch 7 finished I1227 04:35:05.320554 281473083339808 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I1227 04:35:05.321043 281473083339808 failure_handler_test.py:245] Training finished. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 2.22s I1227 04:35:05.322052 281473083339808 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 2.22s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24868 I1227 04:35:05.328748 281473083339808 test_util.py:3999] Using local port 24868 INFO:tensorflow:Using local port 24867 I1227 04:35:05.330414 281473083339808 test_util.py:3999] Using local port 24867 INFO:tensorflow:Using local port 24866 I1227 04:35:05.332016 281473083339808 test_util.py:3999] Using local port 24866 INFO:tensorflow:Using local port 24865 I1227 04:35:05.333508 281473083339808 test_util.py:3999] Using local port 24865 2023-12-27 04:35:06.679358: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2023-12-27 04:35:06.723629: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. INFO:tensorflow:Cluster starting. I1227 04:35:20.626024 281473083339808 failure_handler_test.py:297] Cluster starting. [worker-0]: I1227 04:35:20.700411 281473383298080 multi_process_runner.py:840] Subprocess with PID 560002 (worker, 0) is now being started. [worker-0]: I1227 04:35:20.700791 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24868", "localhost:24867", "localhost:24866", "localhost:24865"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1227 04:35:20.718464 281473383298080 multi_process_runner.py:840] Subprocess with PID 560117 (worker, 1) is now being started. [worker-1]: I1227 04:35:20.718827 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24868", "localhost:24867", "localhost:24866", "localhost:24865"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I1227 04:35:20.748538 281473383298080 multi_process_runner.py:840] Subprocess with PID 560466 (worker, 2) is now being started. [worker-1]: 2023-12-27 04:35:20.752821: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24867 [worker-2]: I1227 04:35:20.748918 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24868", "localhost:24867", "localhost:24866", "localhost:24865"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I1227 04:35:20.775524 281473383298080 multi_process_runner.py:840] Subprocess with PID 560630 (worker, 3) is now being started. [worker-3]: I1227 04:35:20.775889 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24868", "localhost:24867", "localhost:24866", "localhost:24865"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-12-27 04:35:20.783662: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24868 [worker-0]: 2023-12-27 04:35:20.794145: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2044550300334050683 [worker-0]: 2023-12-27 04:35:20.794179: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2023-12-27 04:35:20.794191: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:0 [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:2 [worker-2]: 2023-12-27 04:35:20.797119: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24866 [worker-0]: 2023-12-27 04:35:20.809726: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 16930598770013716507 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:20.809767: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-2]: I0000 00:00:1703651720.810230 560466 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:20.809778: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:0 [worker-0]: /job:worker/replica:0/task:3 [worker-3]: 2023-12-27 04:35:20.815400: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24865 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-1]: I0000 00:00:1703651720.815554 560117 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:20.829367: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 13979573663549427303 [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:20.829405: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-3]: I0000 00:00:1703651720.829622 560630 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:20.829417: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:0 [worker-0]: 2023-12-27 04:35:20.829447: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12473587956212798552 [worker-0]: 2023-12-27 04:35:20.829467: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0000 00:00:1703651720.829628 560002 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-3]: I1227 04:35:20.831367 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I1227 04:35:20.831464 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1227 04:35:20.831862 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1227 04:35:20.846187 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I1227 04:35:20.890438 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-3]: I1227 04:35:20.890415 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-3]: INFO:tensorflow:Check health not enabled. [worker-2]: I1227 04:35:20.891199 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: I1227 04:35:20.890972 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: I1227 04:35:20.890970 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1227 04:35:20.891214 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1227 04:35:20.891213 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1227 04:35:20.891692 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1227 04:35:20.891932 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1227 04:35:20.926871 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1227 04:35:20.927409 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1227 04:35:20.927648 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24868', 'localhost:24867', 'localhost:24866', 'localhost:24865']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1227 04:35:20.967976 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1227 04:35:20.968636 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1227 04:35:20.968879 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: Instructions for updating: [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I1227 04:35:20.969983 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: W1227 04:35:20.969200 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: Instructions for updating: [worker-2]: I1227 04:35:20.970621 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I1227 04:35:20.970866 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I1227 04:35:20.969386 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I1227 04:35:20.969979 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I1227 04:35:20.969914 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-2]: W1227 04:35:20.971206 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: I1227 04:35:20.970572 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-1]: I1227 04:35:20.971560 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1227 04:35:20.970810 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: I1227 04:35:20.971823 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:Start training at 0 [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I1227 04:35:20.971397 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-1]: Instructions for updating: [worker-3]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1227 04:35:20.972145 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: W1227 04:35:20.971138 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-3]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-3]: INFO:tensorflow:Start training at 0 [worker-1]: I1227 04:35:20.972336 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-3]: I1227 04:35:20.971327 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.125664 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.133441 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.133472 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.134580 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.217018 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.216981 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.217263 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.220827 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.285417 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.285480 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.285499 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.285872 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.474019 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.484562 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.502823 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.507354 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.615105 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.645569 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.704782 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.759278 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:21.829347 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1227 04:35:21.829761 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1227 04:35:21.829584 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517f6a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1227 04:35:21.835762 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517f6a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.837856 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.844125 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.849316 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.848116 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517e340> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1227 04:35:21.929504 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517e340> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c900> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:21.929405 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c900> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517f880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1227 04:35:21.934646 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517f880> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:21.938333 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c7c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1227 04:35:21.934827 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c7c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:21.953672 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:21.943422 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:21.961081 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.027266 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.034227 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.027837 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.033161 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.091150 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.091299 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.091340 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.091376 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.149593 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.149785 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.149472 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.149733 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.215521 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.215556 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.220056 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.220180 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.277060 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.277429 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.278326 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.277462 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.342755 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.342841 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.342893 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.342761 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.422653 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.422906 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.423076 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.422882 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.479575 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.479598 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.480347 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.483200 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: I1227 04:35:22.532756 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-0]: I1227 04:35:22.532881 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-2]: I1227 04:35:22.533114 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-1]: I1227 04:35:22.533071 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.541594 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.541762 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.541890 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.542055 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.602169 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.602320 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.606824 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.609918 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.667049 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.670556 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.667253 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.667052 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.740773 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.743551 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.762620 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.770494 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:22.891830 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:22.894867 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:22.909696 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:22.897656 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.018225 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.025381 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.025557 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.025456 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.085221 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.086268 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.088585 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.089627 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1227 04:35:23.745674 281473083339808 failure_handler_test.py:302] sending sigterm [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.174495 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.183334 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.174608 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.174655 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.239141 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.239050 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 29.63s [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1227 04:35:34.954470 281473083339808 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 29.63s [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.239481 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.295453 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.239569 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.295308 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Using local port 24855 I1227 04:35:34.961518 281473083339808 test_util.py:3999] Using local port 24855 [worker-0]: I1227 04:35:23.351327 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.295511 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.296051 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.352224 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.407625 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.352649 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Using local port 24854 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1227 04:35:34.964571 281473083339808 test_util.py:3999] Using local port 24854 [worker-3]: I1227 04:35:23.408526 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.407882 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.464482 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.464704 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.464673 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.352547 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.520498 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.520584 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.521790 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.575649 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.575646 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.407790 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I1227 04:35:23.621847 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.575893 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-3]: I1227 04:35:23.630258 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.465725 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: I1227 04:35:23.622029 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.683794 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.521017 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.630179 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.622215 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.736834 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.575925 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.683506 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.630436 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.790460 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.622179 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.737566 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.683650 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Using local port 24853 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.844973 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1227 04:35:34.977484 281473083339808 test_util.py:3999] Using local port 24853 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.630375 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.790460 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.737229 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.899307 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.683991 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.790786 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Using local port 24852 I1227 04:35:34.980616 281473083339808 test_util.py:3999] Using local port 24852 [worker-0]: I1227 04:35:23.845024 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:23.953737 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.899379 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.737125 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.845192 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.007973 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.790613 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.899574 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:23.953730 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.061911 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:23.953889 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.117594 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.007972 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.845161 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.171535 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.061909 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.008279 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.899442 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.225451 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.061996 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.117760 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:23.953850 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.279470 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.118032 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.171535 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.333416 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.008074 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.171669 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.061931 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.225636 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.387145 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.117893 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: I1227 04:35:24.279631 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.171564 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.433724 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.333561 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.441910 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.225544 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.387294 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.495959 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.279582 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.434064 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-3]: I1227 04:35:24.550555 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.333484 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.604873 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.442117 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.387294 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-2]: I1227 04:35:24.495987 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.434024 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.442021 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.550754 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.658352 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.225469 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.496474 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.712169 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.279471 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.550649 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.604839 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.766961 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.333416 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.604766 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.658590 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.819307 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.387190 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.712421 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.658590 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I1227 04:35:24.871849 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.767318 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.433897 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.712411 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.924479 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.819631 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.441911 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.767247 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:24.976942 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.872211 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.495825 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.819644 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.029468 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.924688 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.081995 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.551057 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.872154 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.134419 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:24.977324 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.924706 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.030226 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.604694 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.188036 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:24.977310 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.082359 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.233324 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-0]: I1227 04:35:24.658427 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.135170 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.029798 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.241043 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.712217 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.188287 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.082279 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.293394 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.345745 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.767139 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.233703 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-1]: I1227 04:35:25.134726 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.400122 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.819458 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.188248 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.241418 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.452687 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: I1227 04:35:24.872001 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.293675 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.233652 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-3]: I1227 04:35:25.504753 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.924541 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.556923 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.346097 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.241280 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.608726 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.293657 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:24.977108 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.660870 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.346177 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.400401 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.029609 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.712953 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.400353 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.453090 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.082096 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.764909 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.453021 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.505070 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.134532 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.816719 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.557235 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.188100 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.505031 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.868703 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.609175 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.233534 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-1]: I1227 04:35:25.557194 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.920388 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.661270 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.241084 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.609123 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:25.972871 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-2]: I1227 04:35:25.713332 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.661147 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.293466 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.019886 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.765322 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.713278 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.345903 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.027491 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.400190 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.765243 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.817042 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.079816 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.452862 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.816959 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.869051 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.131537 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.868940 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.504856 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.183474 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.920735 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.920688 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:25.973202 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.235162 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.557056 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:25.973138 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.020214 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-3]: I1227 04:35:26.287115 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: I1227 04:35:25.608961 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.338725 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.027827 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.020177 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.390492 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.661027 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.027709 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.080196 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.442242 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.080098 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.713092 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.494199 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.131797 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.131821 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.545838 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.765069 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.183750 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.183853 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.597556 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.235475 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.235511 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.651328 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.287433 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.287468 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.703285 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.339018 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.339018 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.390766 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.755838 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.390800 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.442554 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.801921 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-2]: I1227 04:35:26.442628 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.809390 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.494494 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.494574 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.863651 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.546138 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.546151 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.916220 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.597842 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.597956 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.651582 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:26.967719 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.651652 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.703544 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.019750 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.703655 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.756000 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.071213 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.756086 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.802206 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-3]: I1227 04:35:27.123281 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.802428 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-3]: I1227 04:35:27.174821 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.809671 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.226934 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.811395 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.864053 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.816837 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Cluster starting. I1227 04:35:35.105545 281473083339808 failure_handler_test.py:297] Cluster starting. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.280323 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.916522 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.868785 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.864248 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.332245 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.920569 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.916600 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.384556 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:25.972945 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.437292 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.020037 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-3]: I1227 04:35:27.490092 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.027580 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.542734 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-0]: I1227 04:35:26.079979 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.587617 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.131733 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.594949 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:26.968016 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:26.968092 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.183627 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.647240 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.020098 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.020032 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.235299 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.698888 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.071469 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.287345 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.071561 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.751022 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.123590 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.123633 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.802444 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.338810 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.175141 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.175220 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.854609 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.390601 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.227300 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.907786 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.442456 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.227245 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.280668 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.494358 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:27.959950 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.280656 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.546052 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.012112 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.332617 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.332660 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.064375 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.597731 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.384991 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.115776 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.384816 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.651478 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.437708 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.167895 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.437597 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.703439 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.490765 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.219386 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.490381 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:35.156239 281473383298080 multi_process_runner.py:840] Subprocess with PID 587065 (worker, 0) is now being started. [worker-2]: I1227 04:35:27.543189 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.272848 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: I1227 04:35:26.755892 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.543080 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-2]: I1227 04:35:27.587957 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-3]: I1227 04:35:28.324892 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:35.156621 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.587885 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-3]: INFO:tensorflow:epoch 7 finished [worker-2]: I1227 04:35:27.595493 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.802091 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-3]: I1227 04:35:28.369947 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.595198 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.647610 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.809628 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:28.370568 281473383298080 failure_handler_test.py:245] Training finished. [worker-1]: I1227 04:35:27.647578 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.699194 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.863841 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.699208 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.751279 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.751273 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.916405 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.802760 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.802651 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:26.967910 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.854894 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.019912 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.908034 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.071393 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:35.195281 281473383298080 multi_process_runner.py:840] Subprocess with PID 587471 (worker, 1) is now being started. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:27.960200 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.123446 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.854949 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.175021 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.908111 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.012362 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:35.195729 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I1227 04:35:27.227139 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:27.960256 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.064637 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.012623 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.280509 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.116263 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.064698 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.332497 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.168232 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.116144 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.384688 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.437441 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.220028 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.168271 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.490255 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.542870 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: I1227 04:35:28.219701 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.272961 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.273367 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.325721 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.587779 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-2]: I1227 04:35:28.325841 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.595139 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.370260 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-0]: I1227 04:35:27.647415 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:28.371283 281473383298080 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.370319 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-0]: I1227 04:35:27.699141 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:28.371393 281473383298080 failure_handler_test.py:245] Training finished. [worker-0]: I1227 04:35:27.751174 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.802628 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.854754 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.907971 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:27.960046 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.012260 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.064551 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.115999 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.168090 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.219512 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.272852 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:28.325077 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I1227 04:35:28.370137 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I1227 04:35:28.371114 281473383298080 failure_handler_test.py:245] Training finished. [worker-2]: I1227 04:35:35.249619 281473383298080 multi_process_runner.py:840] Subprocess with PID 588022 (worker, 2) is now being started. [worker-0]: 2023-12-27 04:35:35.250573: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24855 [worker-2]: I1227 04:35:35.250102 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: 2023-12-27 04:35:35.258357: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 16231985623393994854 [worker-0]: 2023-12-27 04:35:35.258411: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2023-12-27 04:35:35.258424: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:2 [worker-0]: /job:worker/replica:0/task:1 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1703651735.258647 587065 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-1]: 2023-12-27 04:35:35.266112: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24854 [worker-3]: I1227 04:35:35.278364 281473383298080 multi_process_runner.py:840] Subprocess with PID 588862 (worker, 3) is now being started. [worker-3]: I1227 04:35:35.278811 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2023-12-27 04:35:35.285127: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24853 [worker-0]: 2023-12-27 04:35:35.292476: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 11516134481844973918 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:35.292528: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-2]: I0000 00:00:1703651735.292735 588022 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:35.292540: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:1 [worker-3]: 2023-12-27 04:35:35.314479: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24852 [worker-0]: 2023-12-27 04:35:35.328815: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 15561532833815552042 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:35.328867: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-1]: I0000 00:00:1703651735.329059 587471 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:35.328878: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: 2023-12-27 04:35:35.330722: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 17174492404304306672 [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:35.330768: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-3]: I0000 00:00:1703651735.330960 588862 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I1227 04:35:35.333023 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I1227 04:35:35.332853 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1227 04:35:35.332825 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1227 04:35:35.332722 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1227 04:35:35.403301 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1227 04:35:35.404243 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1227 04:35:35.404489 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I1227 04:35:35.419891 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I1227 04:35:35.421436 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1227 04:35:35.422247 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1227 04:35:35.443803 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1227 04:35:35.444443 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1227 04:35:35.444691 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I1227 04:35:35.463097 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1227 04:35:35.463758 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1227 04:35:35.464006 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1227 04:35:35.740259 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I1227 04:35:35.740430 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1227 04:35:35.742033 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-2]: I1227 04:35:35.741035 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1227 04:35:35.742807 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1227 04:35:35.741304 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: I1227 04:35:35.743516 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1227 04:35:35.759632 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I1227 04:35:35.742331 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: Instructions for updating: [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-1]: I1227 04:35:35.743766 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-2]: W1227 04:35:35.741657 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: I1227 04:35:35.771496 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: W1227 04:35:35.742676 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-2]: INFO:tensorflow:Start training at 0 [worker-1]: W1227 04:35:35.744120 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-2]: I1227 04:35:35.741848 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: I1227 04:35:35.742866 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-1]: I1227 04:35:35.744315 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1227 04:35:35.771917 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1227 04:35:35.772305 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I1227 04:35:35.772503 281473383298080 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:35.967640 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:35.976408 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:35.978136 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.000582 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:36.107953 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:36.116373 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:36.116341 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.125300 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.215871 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:36.221834 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:36.216276 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:36.240624 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.476746 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:36.476880 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:36.481251 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:36.482231 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:36.548521 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:36.565360 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:36.575491 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.579725 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517f920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1227 04:35:36.649937 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517f920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1227 04:35:36.650163 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:36.649796 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c180> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1227 04:35:36.650155 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:36.659547 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:36.660425 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:36.663732 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.680020 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:36.752934 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c360> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1227 04:35:36.753008 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c540> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1227 04:35:36.753286 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1227 04:35:36.753268 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c540> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:36.762689 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:36.763091 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.771995 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:36.790775 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:36.936167 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:36.942008 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:36.946774 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:36.936385 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.025707 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.025967 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.041057 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.049256 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.130441 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.131601 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.137709 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.162580 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.224156 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.224450 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.225568 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.234710 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.313668 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.314953 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.313926 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.318447 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.424417 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.416185 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.434404 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.429569 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.558968 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.559046 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.559575 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.559843 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.620904 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.621169 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.621264 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.620841 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-3]: I1227 04:35:37.674439 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-0]: I1227 04:35:37.674592 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.684579 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-1]: I1227 04:35:37.685654 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-2]: I1227 04:35:37.685859 281473383298080 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.684266 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.695274 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.695781 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.765975 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.765724 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.766048 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.766556 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.831085 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.833893 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.839717 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.840407 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.899264 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.899551 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.899695 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.900068 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:37.961239 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:37.961459 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:37.962100 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:37.962298 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.020507 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.021377 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.020551 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.026514 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.084740 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.084857 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.085515 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.085208 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.145893 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.146364 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.145974 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.147193 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.206959 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.207353 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.206982 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.207574 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.288247 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.294835 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.295491 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.300691 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.364536 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.365153 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.365454 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.370669 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.428896 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.429113 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.428947 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.429349 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.512037 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.512696 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.513080 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.519164 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.586419 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.587378 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.587704 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.588102 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.650529 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.650629 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.666479 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.666623 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I1227 04:35:38.718672 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-0]: I1227 04:35:38.718905 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: I1227 04:35:38.719072 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-1]: I1227 04:35:38.719201 281473383298080 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.729165 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.729199 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.729772 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.749654 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.810981 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.811054 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.811923 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.815778 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:38.921640 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:38.949944 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:38.953678 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:38.950885 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:39.076517 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:39.076731 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:39.078412 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:39.078622 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:39.276991 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I1227 04:35:39.295504 281473083339808 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I1227 04:35:39.295922 281473083339808 failure_handler_test.py:306] sigterm sent [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:39.279604 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:39.281694 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-0]: I1227 04:35:39.294156 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:39.297509 281473383298080 failure_handling.py:710] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I1227 04:35:39.678955 281473383298080 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:39.683075 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-0]: I1227 04:35:39.684748 281447965127136 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:39.684098 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:39.688018 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-3]: I1227 04:35:39.689801 281447562473952 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-1]: I1227 04:35:39.696866 281448971760096 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 36 [worker-2]: I1227 04:35:39.705551 281473383298080 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 36 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I1227 04:35:39.705828 281448292217312 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-2]: I1227 04:35:39.725218 281473383298080 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I1227 04:35:39.732558 281473383298080 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I1227 04:35:39.733466 281473383298080 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I1227 04:35:39.734187 281473383298080 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:39.743682 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I1227 04:35:39.806827 281473383298080 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I1227 04:35:39.815917 281473383298080 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I1227 04:35:39.806116 281473383298080 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I1227 04:35:39.817038 281473383298080 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/workertemp_3/fh_ckpt [worker-3]: I1227 04:35:39.868590 281473383298080 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/workertemp_3/fh_ckpt [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/fh_ckpt [worker-0]: I1227 04:35:39.873431 281473383298080 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/fh_ckpt [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/workertemp_1/fh_ckpt [worker-1]: I1227 04:35:39.877717 281473383298080 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/workertemp_1/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I1227 04:35:39.878978 281473383298080 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: I1227 04:35:39.879161 281473383298080 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I1227 04:35:39.879206 281473383298080 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: I1227 04:35:39.879160 281473383298080 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I1227 04:35:39.879396 281473383298080 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I1227 04:35:39.879493 281473383298080 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/workertemp_2/fh_ckpt [worker-2]: I1227 04:35:39.949898 281473383298080 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2339by0tc/tmpdu6_flt1/workertemp_2/fh_ckpt [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: I1227 04:35:39.951246 281473383298080 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I1227 04:35:39.951434 281473383298080 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I1227 04:35:41.296669 281473083339808 failure_handler_test.py:309] restarting workers [worker-0]: I1227 04:35:41.343359 281473383298080 multi_process_runner.py:840] Subprocess with PID 652523 (worker, 0) is now being started. [worker-0]: I1227 04:35:41.343797 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1227 04:35:41.365297 281473383298080 multi_process_runner.py:840] Subprocess with PID 652920 (worker, 1) is now being started. INFO:tensorflow:workers restarted I1227 04:35:41.367767 281473083339808 failure_handler_test.py:313] workers restarted [worker-1]: I1227 04:35:41.365715 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2023-12-27 04:35:41.394141: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24855 [worker-0]: 2023-12-27 04:35:41.397581: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 2190594846865204621 [worker-0]: 2023-12-27 04:35:41.397619: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 3/4 tasks to connect. [worker-0]: 2023-12-27 04:35:41.397630: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:2 [worker-0]: /job:worker/replica:0/task:1 [worker-0]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: I0000 00:00:1703651741.398077 652523 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-1]: 2023-12-27 04:35:41.421116: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24854 [worker-2]: I1227 04:35:41.437799 281473383298080 multi_process_runner.py:840] Subprocess with PID 653137 (worker, 2) is now being started. [worker-2]: I1227 04:35:41.438253 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I1227 04:35:41.439489 281473383298080 multi_process_runner.py:840] Subprocess with PID 653493 (worker, 3) is now being started. [worker-3]: I1227 04:35:41.439932 281473383298080 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24855", "localhost:24854", "localhost:24853", "localhost:24852"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-12-27 04:35:41.479401: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 6098562064255592375 [worker-1]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:41.479448: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 2/4 tasks to connect. [worker-1]: I0000 00:00:1703651741.479624 652920 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-0]: 2023-12-27 04:35:41.479459: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:3 [worker-0]: /job:worker/replica:0/task:2 [worker-3]: 2023-12-27 04:35:41.492427: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24852 [worker-0]: 2023-12-27 04:35:41.502895: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 10558448655264028083 [worker-0]: 2023-12-27 04:35:41.502936: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 1/4 tasks to connect. [worker-0]: 2023-12-27 04:35:41.502948: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:542] Example stragglers: [worker-0]: /job:worker/replica:0/task:2 [worker-3]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-3]: I0000 00:00:1703651741.504005 653493 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-2]: 2023-12-27 04:35:41.513477: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24853 [worker-0]: 2023-12-27 04:35:41.523446: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:577] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 13385251884878560877 [worker-2]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR [worker-0]: 2023-12-27 04:35:41.523503: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:539] Waiting for 0/4 tasks to connect. [worker-2]: I0000 00:00:1703651741.523719 653137 coordination_service_agent.cc:309] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I1227 04:35:41.525742 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I1227 04:35:41.525745 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1227 04:35:41.525822 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1227 04:35:41.526119 281473383298080 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-3]: I1227 04:35:41.588555 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I1227 04:35:41.588604 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1227 04:35:41.589434 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: I1227 04:35:41.589181 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1227 04:35:41.589686 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1227 04:35:41.589427 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1227 04:35:41.599400 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1227 04:35:41.600043 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1227 04:35:41.600285 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1227 04:35:41.628614 281473383298080 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1227 04:35:41.629270 281473383298080 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1227 04:35:41.629523 281473383298080 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24855', 'localhost:24854', 'localhost:24853', 'localhost:24852']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1227 04:35:41.742420 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I1227 04:35:41.742514 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1227 04:35:41.743454 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-0]: I1227 04:35:41.743491 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1227 04:35:41.743766 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: I1227 04:35:41.743736 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W1227 04:35:41.744082 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 36 [worker-0]: W1227 04:35:41.744122 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I1227 04:35:41.744277 281473383298080 failure_handler_test.py:197] Start training at 36 [worker-0]: Instructions for updating: [worker-2]: INFO:tensorflow:training restarted [worker-2]: I1227 04:35:41.749152 281473383298080 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I1227 04:35:41.749963 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I1227 04:35:41.750677 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1227 04:35:41.750934 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1227 04:35:41.751271 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 36 [worker-0]: I1227 04:35:41.744317 281473383298080 failure_handler_test.py:197] Start training at 36 [worker-0]: INFO:tensorflow:training restarted [worker-0]: I1227 04:35:41.749549 281473383298080 failure_handler_test.py:207] training restarted [worker-1]: I1227 04:35:41.750025 281473383298080 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start training at 36 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I1227 04:35:41.751462 281473383298080 failure_handler_test.py:197] Start training at 36 [worker-1]: I1227 04:35:41.750768 281473383298080 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:training restarted [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1227 04:35:41.754570 281473383298080 failure_handler_test.py:207] training restarted [worker-1]: I1227 04:35:41.751036 281473383298080 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1227 04:35:41.751405 281473383298080 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 36 [worker-1]: I1227 04:35:41.751604 281473383298080 failure_handler_test.py:197] Start training at 36 [worker-1]: INFO:tensorflow:training restarted [worker-1]: I1227 04:35:41.754893 281473383298080 failure_handler_test.py:207] training restarted [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:41.913196 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:41.938267 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:41.939082 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:42.088930 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:42.165700 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:42.166246 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:42.170866 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:42.166559 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:42.391719 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:42.440426 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:42.441714 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:42.567271 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:42.651667 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:42.660057 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:42.670524 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:42.693504 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:42.822557 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:42.822422 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:42.850112 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:42.839828 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffef517c0e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1227 04:35:42.972377 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c0e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1227 04:35:42.966451 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:42.955132 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c400> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:42.976851 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:42.985714 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W1227 04:35:42.971644 281473383298080 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:43.025913 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:43.089572 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517dc60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1227 04:35:43.146391 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517dc60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517cf40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1227 04:35:43.155750 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517cf40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: W1227 04:35:43.156776 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffef517c4a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:43.165812 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:43.165923 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:43.155243 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W1227 04:35:43.160911 281473383298080 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xfffef517c4a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:43.192450 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:43.304410 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:43.324446 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:43.342809 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:43.340267 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:43.480732 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:43.490206 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:43.499873 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:43.519437 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I1227 04:35:43.638581 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I1227 04:35:43.647100 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:43.648633 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-3]: I1227 04:35:43.648711 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:43.657157 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:43.658222 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I1227 04:35:43.665346 281473383298080 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:43.675608 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:43.736972 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:43.739282 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:43.763197 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:43.777083 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:43.887073 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:43.900766 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:43.895925 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:43.914603 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.047517 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.048259 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.049133 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.078372 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.138545 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.143650 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.144476 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.145455 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.210719 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.211853 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.211990 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.229850 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.291703 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.293456 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.304165 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.304499 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.374503 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.375807 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.381996 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.385914 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.490689 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.499940 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.520313 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.499387 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.604582 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.610367 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.610099 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.620791 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.799839 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.825905 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.829381 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.829843 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.893108 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.893025 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.892814 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.893114 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:44.965798 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:44.961152 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:44.961527 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:44.961765 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.031718 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.031905 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.032485 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.032322 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.149431 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.150003 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.150119 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.151152 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: I1227 04:35:45.201417 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-0]: I1227 04:35:45.201552 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-1]: I1227 04:35:45.201699 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-2]: I1227 04:35:45.201739 281473383298080 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.210731 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.210842 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.210984 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.211445 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.274293 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.274789 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.277879 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.280366 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.371850 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.372566 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.372954 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.372961 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.483728 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.493840 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.496615 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.510062 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.598509 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.598966 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.617134 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.599190 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.686913 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.690364 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.694461 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.699231 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.758894 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.759099 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.769837 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.774511 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.834519 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.835725 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.844717 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.835093 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.934824 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:45.935454 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.935066 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.936184 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:45.996312 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:45.997416 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:45.996865 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.039534 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.104468 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.112666 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.118148 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.130195 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.212293 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.212453 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.212693 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.212919 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.288731 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.294007 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.295617 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.309665 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.394919 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.401665 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.410206 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.423604 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.513748 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.514884 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.524508 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.519961 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I1227 04:35:46.611276 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: I1227 04:35:46.615707 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I1227 04:35:46.616054 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-1]: I1227 04:35:46.615961 281473383298080 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.627372 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.627831 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.625488 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.620745 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.796231 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.795644 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.810607 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.812085 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.892678 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.892851 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.892651 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.894001 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:46.954273 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:46.954877 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:46.954381 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:46.955408 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.042833 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.049272 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.090430 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.093462 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.164495 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.164433 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.165264 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.170169 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.229036 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.229451 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.229851 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.230087 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.298193 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.299149 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.299270 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.302803 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.365053 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.365337 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.365799 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.370441 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.438080 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.439143 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.448120 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.440420 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.535467 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.536787 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.539456 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.536102 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.600494 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.600708 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.601225 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.603774 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.703997 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.705746 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.733195 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.725704 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.807010 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.807468 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.806576 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.826014 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.895391 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.900123 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.904560 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.912819 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-3]: I1227 04:35:47.964452 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-0]: I1227 04:35:47.964637 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-2]: I1227 04:35:47.964878 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-1]: I1227 04:35:47.964837 281473383298080 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:47.973707 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:47.975013 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:47.975416 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:47.979109 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.039183 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.040539 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.040384 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.040992 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.115335 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.115155 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.115397 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.115686 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.184416 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.188466 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.185898 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.185241 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.264271 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.264884 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.267672 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.265286 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.330318 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.344226 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.349098 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.349508 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.435328 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.441370 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.452073 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.486062 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.565495 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.565936 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.566383 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.565436 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.633790 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.639283 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.659997 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.669719 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.779995 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.778056 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.789358 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.809957 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:48.906477 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:48.914435 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:48.914818 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:48.929716 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.021522 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.021676 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:49.022073 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:49.044538 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.105007 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:49.110867 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:49.110996 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.140376 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.212290 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:49.214575 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.217288 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:49.227795 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.294811 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.295230 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:49.295690 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:49.295410 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-3]: I1227 04:35:49.345215 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-1]: I1227 04:35:49.345557 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-0]: I1227 04:35:49.345381 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-2]: I1227 04:35:49.345603 281473383298080 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.489904 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.500738 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:49.590190 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:49.581372 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.690371 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.696309 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:49.696507 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:49.719837 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:49.802162 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.802179 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.802076 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:49.860609 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:49.987145 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:49.998039 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.010570 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.002390 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.080891 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.081327 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.099724 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.100655 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.185851 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.186398 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.189694 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.185983 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.276060 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.276138 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.276141 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.283896 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.356594 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.358488 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.368164 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.378127 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.524518 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.530232 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.547344 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.570091 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.686171 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.689565 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.699285 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.729753 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.859378 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.879634 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.869502 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.879655 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:50.958266 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:50.959126 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:50.958602 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:50.970098 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:51.051567 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:51.051827 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:51.052310 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:51.052286 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:51.115362 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:51.124852 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:51.159932 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:51.159832 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1227 04:35:51.265664 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1227 04:35:51.270521 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1227 04:35:51.265664 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1227 04:35:51.292895 281473383298080 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:epoch 7 finished [worker-3]: I1227 04:35:51.357042 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-1]: I1227 04:35:51.359267 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I1227 04:35:51.360304 281473383298080 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I1227 04:35:51.366486 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-2]: I1227 04:35:51.368170 281473383298080 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I1227 04:35:51.369033 281473383298080 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I1227 04:35:51.384104 281473383298080 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I1227 04:35:51.385875 281473383298080 failure_handler_test.py:245] Training finished. I1227 04:35:52.369210 281473083339808 multi_process_runner.py:646] worker-0 exit code: 0 I1227 04:35:52.369485 281473083339808 multi_process_runner.py:646] worker-1 exit code: 0 I1227 04:35:52.369639 281473083339808 multi_process_runner.py:646] worker-2 exit code: 0 I1227 04:35:52.369783 281473083339808 multi_process_runner.py:646] worker-3 exit code: 0 I1227 04:35:52.372031 281473083339808 multi_process_runner.py:662] Joining log reading threads. I1227 04:35:52.372308 281473083339808 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 17.66s I1227 04:35:52.619368 281473083339808 test_util.py:2620] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 17.66s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker ====================================================================== ERROR: test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker(api_wrapping_train=False, input_arg='checkpoint', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process ---------------------------------------------------------------------- Ran 3 tests in 49.520s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/core/grappler/optimizers:mkl_remapper_test: 2023-12-27 04:37:54.384714: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [==========] Running 81 tests from 9 test suites. [----------] Global test environment set-up. [----------] 54 tests from MklRemapperTest [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu_addbcastfalse (259 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu6_addbcastfalse (249 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationElu_addbcastfalse (311 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationLeakyRelu_addbcastfalse (169 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddN" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse (120 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu_addbcastfalse (219 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu6_addbcastfalse (221 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationElu_addbcastfalse (151 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationLeakyRelu_addbcastfalse (145 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddN" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse (59 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcastfalse (110 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcastfalse (141 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcastfalse (113 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcastfalse (115 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse (60 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcastfalse (155 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcastfalse (159 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcastfalse (115 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcastfalse (118 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse (59 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcasttrue (162 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcasttrue (379 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcasttrue (789 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcasttrue (241 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcasttrue (227 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcasttrue (178 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcasttrue (111 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcasttrue (759 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcasttrue (557 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcasttrue (45 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcastfalse (169 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcastfalse (196 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcastfalse (219 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcastfalse (141 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddV2" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse (59 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcastfalse (105 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcastfalse (149 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcastfalse (135 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcastfalse (215 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddV2" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse (59 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcasttrue (48 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcasttrue (69 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcasttrue (80 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcasttrue (59 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcasttrue (68 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcasttrue (126 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcasttrue (119 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcasttrue (89 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcasttrue (70 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcasttrue (87 ms) [ RUN ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.op() Which is: "BiasAdd" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.input_size() Which is: 2 3 [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT (2 ms) [ RUN ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.op() Which is: "BiasAdd" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.input_size() Which is: 2 3 [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 (2 ms) [ RUN ] MklRemapperTest.FuseBatchNormWithRelu [ OK ] MklRemapperTest.FuseBatchNormWithRelu (123 ms) [ RUN ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:514: Failure Expected equality of these values: "_FusedMatMul" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:515: Failure Expected equality of these values: "input" node.input(0) Which is: "bias_add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:516: Failure Expected equality of these values: "filter" node.input(1) Which is: "input_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd (1 ms) [----------] 54 tests from MklRemapperTest (8936 ms total) [----------] 2 tests from RelpaceAddWithBiasAddTest [ RUN ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:621: Failure Expected equality of these values: node.op() Which is: "AddV2" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:622: Failure Expected: (node.input_size()) >= (3), actual: 2 vs 3 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 (2 ms) [ RUN ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:621: Failure Expected equality of these values: node.op() Which is: "Add" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:622: Failure Expected: (node.input_size()) >= (3), actual: 2 vs 3 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add (2 ms) [----------] 2 tests from RelpaceAddWithBiasAddTest (4 ms total) [----------] 4 tests from FusedMatMulBiasAddAndGeluTest [ RUN ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact [ OK ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact (12 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:768: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedMatMulBiasAddAndGelu with bfloat16. [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact (0 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact2 [ OK ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact2 (22 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:777: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedMatMulBiasAddAndGelu with bfloat16. [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 (0 ms) [----------] 4 tests from FusedMatMulBiasAddAndGeluTest (34 ms total) [----------] 2 tests from MklFusedBatchMatMul [ RUN ] MklFusedBatchMatMul.MulAndAdd tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd (201 ms) [ RUN ] MklFusedBatchMatMul.MulAndAdd2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd2 (238 ms) [----------] 2 tests from MklFusedBatchMatMul (440 ms total) [----------] 2 tests from MklRemapperSwishTest [ RUN ] MklRemapperSwishTest.F32 [ OK ] MklRemapperSwishTest.F32 (111 ms) [ RUN ] MklRemapperSwishTest.BF16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1083: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklRemapperSwish with bfloat16. [ SKIPPED ] MklRemapperSwishTest.BF16 (0 ms) [----------] 2 tests from MklRemapperSwishTest (111 ms total) [----------] 2 tests from MklRemapperConv2dBiasAddSwishTest [ RUN ] MklRemapperConv2dBiasAddSwishTest.F32 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1138: Failure Expected equality of these values: node.op() Which is: "_MklSwish" "_FusedConv2D" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1139: Failure Expected equality of these values: node.input_size() Which is: 1 3 [ FAILED ] MklRemapperConv2dBiasAddSwishTest.F32 (1 ms) [ RUN ] MklRemapperConv2dBiasAddSwishTest.BF16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1169: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklRemapperConv2dBiasAddSwish with bfloat16. [ SKIPPED ] MklRemapperConv2dBiasAddSwishTest.BF16 (0 ms) [----------] 2 tests from MklRemapperConv2dBiasAddSwishTest (1 ms total) [----------] 1 test from MklRemapperConv2dFusedBatchNormSwishTest [ RUN ] MklRemapperConv2dFusedBatchNormSwishTest.F32 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1234: Failure Expected equality of these values: node.op() Which is: "_MklSwish" "_FusedConv2D" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1235: Failure Expected equality of these values: node.input_size() Which is: 1 6 [ FAILED ] MklRemapperConv2dFusedBatchNormSwishTest.F32 (1 ms) [----------] 1 test from MklRemapperConv2dFusedBatchNormSwishTest (1 ms total) [----------] 8 tests from MklFuseInstanceNormTest [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NDHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NDHWC (147 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NCDHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NCDHW (66 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NHWC (126 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NCHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NCHW (97 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NDHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NDHWC (236 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NCDHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NCDHW (127 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NHWC (92 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NCHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NCHW (119 ms) [----------] 8 tests from MklFuseInstanceNormTest (1013 ms total) [----------] 6 tests from FusedConvBiasAddAndHardSwishTest [ RUN ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast (0 ms) [----------] 6 tests from FusedConvBiasAddAndHardSwishTest (0 ms total) [----------] Global test environment tear-down [==========] 81 tests from 9 test suites ran. (10545 ms total) [ PASSED ] 56 tests. [ SKIPPED ] 12 tests, listed below: [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd2 [ SKIPPED ] MklRemapperSwishTest.BF16 [ SKIPPED ] MklRemapperConv2dBiasAddSwishTest.BF16 [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast [ FAILED ] 13 tests, listed below: [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 [ FAILED ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add [ FAILED ] MklRemapperConv2dBiasAddSwishTest.F32 [ FAILED ] MklRemapperConv2dFusedBatchNormSwishTest.F32 13 FAILED TESTS ================================================================================ ==================== Test output for //tensorflow/core/grappler/optimizers:mkl_remapper_test: 2023-12-27 04:38:08.411370: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [==========] Running 81 tests from 9 test suites. [----------] Global test environment set-up. [----------] 54 tests from MklRemapperTest [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu_addbcastfalse (339 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu6_addbcastfalse (253 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationElu_addbcastfalse (116 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationLeakyRelu_addbcastfalse (134 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddN" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse (74 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu_addbcastfalse (166 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu6_addbcastfalse (211 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationElu_addbcastfalse (239 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationLeakyRelu_addbcastfalse (136 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddN" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse (98 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcastfalse (122 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcastfalse (161 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcastfalse (154 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcastfalse (245 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse (209 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcastfalse (205 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcastfalse (169 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcastfalse (176 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcastfalse (195 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse (228 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcasttrue (55 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcasttrue (44 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcasttrue (80 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcasttrue (127 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcasttrue (292 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcasttrue (86 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcasttrue (194 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcasttrue (139 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcasttrue (112 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcasttrue (100 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcastfalse (201 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcastfalse (182 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcastfalse (239 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcastfalse (165 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddV2" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse (159 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcastfalse (225 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcastfalse (144 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcastfalse (200 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcastfalse (147 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddV2" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse (59 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcasttrue (31 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcasttrue (54 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcasttrue (50 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcasttrue (55 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcasttrue (79 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcasttrue (135 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcasttrue (67 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcasttrue (73 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcasttrue (101 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcasttrue (77 ms) [ RUN ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.op() Which is: "BiasAdd" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.input_size() Which is: 2 3 [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT (2 ms) [ RUN ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.op() Which is: "BiasAdd" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.input_size() Which is: 2 3 [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 (2 ms) [ RUN ] MklRemapperTest.FuseBatchNormWithRelu [ OK ] MklRemapperTest.FuseBatchNormWithRelu (140 ms) [ RUN ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:514: Failure Expected equality of these values: "_FusedMatMul" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:515: Failure Expected equality of these values: "input" node.input(0) Which is: "bias_add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:516: Failure Expected equality of these values: "filter" node.input(1) Which is: "input_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd (1 ms) [----------] 54 tests from MklRemapperTest (7804 ms total) [----------] 2 tests from RelpaceAddWithBiasAddTest [ RUN ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:621: Failure Expected equality of these values: node.op() Which is: "AddV2" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:622: Failure Expected: (node.input_size()) >= (3), actual: 2 vs 3 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 (2 ms) [ RUN ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:621: Failure Expected equality of these values: node.op() Which is: "Add" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:622: Failure Expected: (node.input_size()) >= (3), actual: 2 vs 3 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add (2 ms) [----------] 2 tests from RelpaceAddWithBiasAddTest (4 ms total) [----------] 4 tests from FusedMatMulBiasAddAndGeluTest [ RUN ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact [ OK ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact (77 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:768: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedMatMulBiasAddAndGelu with bfloat16. [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact (0 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact2 [ OK ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact2 (37 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:777: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedMatMulBiasAddAndGelu with bfloat16. [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 (0 ms) [----------] 4 tests from FusedMatMulBiasAddAndGeluTest (118 ms total) [----------] 2 tests from MklFusedBatchMatMul [ RUN ] MklFusedBatchMatMul.MulAndAdd tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd (166 ms) [ RUN ] MklFusedBatchMatMul.MulAndAdd2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd2 (172 ms) [----------] 2 tests from MklFusedBatchMatMul (338 ms total) [----------] 2 tests from MklRemapperSwishTest [ RUN ] MklRemapperSwishTest.F32 [ OK ] MklRemapperSwishTest.F32 (60 ms) [ RUN ] MklRemapperSwishTest.BF16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1083: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklRemapperSwish with bfloat16. [ SKIPPED ] MklRemapperSwishTest.BF16 (0 ms) [----------] 2 tests from MklRemapperSwishTest (60 ms total) [----------] 2 tests from MklRemapperConv2dBiasAddSwishTest [ RUN ] MklRemapperConv2dBiasAddSwishTest.F32 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1138: Failure Expected equality of these values: node.op() Which is: "_MklSwish" "_FusedConv2D" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1139: Failure Expected equality of these values: node.input_size() Which is: 1 3 [ FAILED ] MklRemapperConv2dBiasAddSwishTest.F32 (1 ms) [ RUN ] MklRemapperConv2dBiasAddSwishTest.BF16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1169: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklRemapperConv2dBiasAddSwish with bfloat16. [ SKIPPED ] MklRemapperConv2dBiasAddSwishTest.BF16 (0 ms) [----------] 2 tests from MklRemapperConv2dBiasAddSwishTest (1 ms total) [----------] 1 test from MklRemapperConv2dFusedBatchNormSwishTest [ RUN ] MklRemapperConv2dFusedBatchNormSwishTest.F32 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1234: Failure Expected equality of these values: node.op() Which is: "_MklSwish" "_FusedConv2D" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1235: Failure Expected equality of these values: node.input_size() Which is: 1 6 [ FAILED ] MklRemapperConv2dFusedBatchNormSwishTest.F32 (1 ms) [----------] 1 test from MklRemapperConv2dFusedBatchNormSwishTest (1 ms total) [----------] 8 tests from MklFuseInstanceNormTest [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NDHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NDHWC (92 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NCDHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NCDHW (64 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NHWC (60 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NCHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NCHW (41 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NDHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NDHWC (92 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NCDHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NCDHW (75 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NHWC (117 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NCHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NCHW (94 ms) [----------] 8 tests from MklFuseInstanceNormTest (640 ms total) [----------] 6 tests from FusedConvBiasAddAndHardSwishTest [ RUN ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast (0 ms) [----------] 6 tests from FusedConvBiasAddAndHardSwishTest (0 ms total) [----------] Global test environment tear-down [==========] 81 tests from 9 test suites ran. (8970 ms total) [ PASSED ] 56 tests. [ SKIPPED ] 12 tests, listed below: [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd2 [ SKIPPED ] MklRemapperSwishTest.BF16 [ SKIPPED ] MklRemapperConv2dBiasAddSwishTest.BF16 [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast [ FAILED ] 13 tests, listed below: [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 [ FAILED ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add [ FAILED ] MklRemapperConv2dBiasAddSwishTest.F32 [ FAILED ] MklRemapperConv2dFusedBatchNormSwishTest.F32 13 FAILED TESTS ================================================================================ ==================== Test output for //tensorflow/core/grappler/optimizers:mkl_remapper_test: 2023-12-27 04:38:19.769383: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [==========] Running 81 tests from 9 test suites. [----------] Global test environment set-up. [----------] 54 tests from MklRemapperTest [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu_addbcastfalse (319 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationRelu6_addbcastfalse (163 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationElu_addbcastfalse (274 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationLeakyRelu_addbcastfalse (250 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddN" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse (160 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu_addbcastfalse (203 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationRelu6_addbcastfalse (207 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationElu_addbcastfalse (198 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationLeakyRelu_addbcastfalse (189 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddN" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse (59 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcastfalse (96 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcastfalse (121 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcastfalse (172 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcastfalse (243 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse (109 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcastfalse (165 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcastfalse (148 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcastfalse (278 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcastfalse (354 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse (125 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu_addbcasttrue (76 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationRelu6_addbcasttrue (50 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationElu_addbcasttrue (31 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationLeakyRelu_addbcasttrue (32 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcasttrue (116 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu_addbcasttrue (95 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationRelu6_addbcasttrue (46 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationElu_addbcasttrue (76 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationLeakyRelu_addbcasttrue (99 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcasttrue (65 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcastfalse (104 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcastfalse (108 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcastfalse (93 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcastfalse (90 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddV2" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse (79 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcastfalse (179 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcastfalse (136 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcastfalse (189 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcastfalse [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcastfalse (111 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:141: Failure Expected equality of these values: "_FusedConv2D" node.op() Which is: "AddV2" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:142: Failure Expected equality of these values: "input" node.input(0) Which is: "input_addn" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:143: Failure Expected equality of these values: "filter" node.input(1) Which is: "bias_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse (61 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu_addbcasttrue (37 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationRelu6_addbcasttrue (34 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationElu_addbcasttrue (82 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationLeakyRelu_addbcasttrue (108 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcasttrue (51 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu_addbcasttrue (64 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationRelu6_addbcasttrue (46 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationElu_addbcasttrue (74 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationLeakyRelu_addbcasttrue (77 ms) [ RUN ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcasttrue [ OK ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcasttrue (117 ms) [ RUN ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.op() Which is: "BiasAdd" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.input_size() Which is: 2 3 [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT (2 ms) [ RUN ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.op() Which is: "BiasAdd" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:311: Failure Expected equality of these values: node.input_size() Which is: 2 3 [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 (2 ms) [ RUN ] MklRemapperTest.FuseBatchNormWithRelu [ OK ] MklRemapperTest.FuseBatchNormWithRelu (151 ms) [ RUN ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:514: Failure Expected equality of these values: "_FusedMatMul" node.op() Which is: "Add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:515: Failure Expected equality of these values: "input" node.input(0) Which is: "bias_add" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:516: Failure Expected equality of these values: "filter" node.input(1) Which is: "input_add" [libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/map.h:1293] CHECK failed: it != end(): key not found: num_args unknown file: Failure C++ exception with description "CHECK failed: it != end(): key not found: num_args" thrown in the test body. [ FAILED ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd (1 ms) [----------] 54 tests from MklRemapperTest (6706 ms total) [----------] 2 tests from RelpaceAddWithBiasAddTest [ RUN ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:621: Failure Expected equality of these values: node.op() Which is: "AddV2" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:622: Failure Expected: (node.input_size()) >= (3), actual: 2 vs 3 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 (2 ms) [ RUN ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:621: Failure Expected equality of these values: node.op() Which is: "Add" "_FusedDepthwiseConv2dNative" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:622: Failure Expected: (node.input_size()) >= (3), actual: 2 vs 3 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add (2 ms) [----------] 2 tests from RelpaceAddWithBiasAddTest (4 ms total) [----------] 4 tests from FusedMatMulBiasAddAndGeluTest [ RUN ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact [ OK ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact (15 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:768: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedMatMulBiasAddAndGelu with bfloat16. [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact (0 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact2 [ OK ] FusedMatMulBiasAddAndGeluTest.Float32GeluExact2 (29 ms) [ RUN ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:777: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedMatMulBiasAddAndGelu with bfloat16. [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 (0 ms) [----------] 4 tests from FusedMatMulBiasAddAndGeluTest (54 ms total) [----------] 2 tests from MklFusedBatchMatMul [ RUN ] MklFusedBatchMatMul.MulAndAdd tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:788: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd (89 ms) [ RUN ] MklFusedBatchMatMul.MulAndAdd2 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:883: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklFusedBatchMatMul with bfloat16. [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd2 (89 ms) [----------] 2 tests from MklFusedBatchMatMul (178 ms total) [----------] 2 tests from MklRemapperSwishTest [ RUN ] MklRemapperSwishTest.F32 [ OK ] MklRemapperSwishTest.F32 (22 ms) [ RUN ] MklRemapperSwishTest.BF16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1083: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklRemapperSwish with bfloat16. [ SKIPPED ] MklRemapperSwishTest.BF16 (0 ms) [----------] 2 tests from MklRemapperSwishTest (23 ms total) [----------] 2 tests from MklRemapperConv2dBiasAddSwishTest [ RUN ] MklRemapperConv2dBiasAddSwishTest.F32 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1138: Failure Expected equality of these values: node.op() Which is: "_MklSwish" "_FusedConv2D" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1139: Failure Expected equality of these values: node.input_size() Which is: 1 3 [ FAILED ] MklRemapperConv2dBiasAddSwishTest.F32 (1 ms) [ RUN ] MklRemapperConv2dBiasAddSwishTest.BF16 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1169: Skipped Intel oneDNN with bfloat16 is not supported, skipping MklRemapperConv2dBiasAddSwish with bfloat16. [ SKIPPED ] MklRemapperConv2dBiasAddSwishTest.BF16 (0 ms) [----------] 2 tests from MklRemapperConv2dBiasAddSwishTest (1 ms total) [----------] 1 test from MklRemapperConv2dFusedBatchNormSwishTest [ RUN ] MklRemapperConv2dFusedBatchNormSwishTest.F32 tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1234: Failure Expected equality of these values: node.op() Which is: "_MklSwish" "_FusedConv2D" tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1235: Failure Expected equality of these values: node.input_size() Which is: 1 6 [ FAILED ] MklRemapperConv2dFusedBatchNormSwishTest.F32 (1 ms) [----------] 1 test from MklRemapperConv2dFusedBatchNormSwishTest (1 ms total) [----------] 8 tests from MklFuseInstanceNormTest [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NDHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NDHWC (35 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NCDHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm5D_FP32_NCDHW (21 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NHWC (25 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NCHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNorm4D_FP32_NCHW (27 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NDHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NDHWC (112 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NCDHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation5D_FP32_NCDHW (43 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NHWC [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NHWC (48 ms) [ RUN ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NCHW [ OK ] MklFuseInstanceNormTest.FuseMklInstanceNormWithActivation4D_FP32_NCHW (45 ms) [----------] 8 tests from MklFuseInstanceNormTest (981 ms total) [----------] 6 tests from FusedConvBiasAddAndHardSwishTest [ RUN ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast (0 ms) [ RUN ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast tensorflow/core/grappler/optimizers/mkl_remapper_test.cc:1563: Skipped Intel oneDNN with bfloat16 is not supported, skipping FusedConvBiasAddAndHardSwishTest with bfloat16. [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast (0 ms) [----------] 6 tests from FusedConvBiasAddAndHardSwishTest (0 ms total) [----------] Global test environment tear-down [==========] 81 tests from 9 test suites ran. (7952 ms total) [ PASSED ] 56 tests. [ SKIPPED ] 12 tests, listed below: [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact [ SKIPPED ] FusedMatMulBiasAddAndGeluTest.BFloat16GeluExact2 [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd [ SKIPPED ] MklFusedBatchMatMul.MulAndAdd2 [ SKIPPED ] MklRemapperSwishTest.BF16 [ SKIPPED ] MklRemapperConv2dBiasAddSwishTest.BF16 [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32Conv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Float32DWConv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwish [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16Conv2DBiasHardSwishWithCast [ SKIPPED ] FusedConvBiasAddAndHardSwishTest.Bfloat16DWConv2DBiasHardSwishWithCast [ FAILED ] 13 tests, listed below: [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddN_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAdd_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NHWC_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseConv2DWithBiasAndAddV2_NCHW_activationNone_addbcastfalse [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_FLOAT [ FAILED ] MklRemapperTest.FuseDepthwiseConv2DWithBiasAndActivation_DT_BFLOAT16 [ FAILED ] MklRemapperTest.FuseMatMulWithBiasAddAndAdd [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_AddV2 [ FAILED ] RelpaceAddWithBiasAddTest.RelpaceAddWithBiasAddDepthConv2D_Add [ FAILED ] MklRemapperConv2dBiasAddSwishTest.F32 [ FAILED ] MklRemapperConv2dFusedBatchNormSwishTest.F32 13 FAILED TESTS ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 30.9s //tensorflow/c:c_api_function_test PASSED in 40.8s //tensorflow/c:c_api_test_cpu PASSED in 42.3s //tensorflow/c:c_test PASSED in 29.4s //tensorflow/c:env_test_cpu PASSED in 24.4s //tensorflow/c:kernels_test_cpu PASSED in 35.6s //tensorflow/c:ops_test PASSED in 21.9s //tensorflow/c:tf_status_helper_test PASSED in 0.1s //tensorflow/c:while_loop_test PASSED in 31.2s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 45.7s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 48.7s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 29.0s //tensorflow/c/eager:c_api_test_cpu PASSED in 41.1s //tensorflow/c/eager:custom_device_test PASSED in 29.6s //tensorflow/c/eager:dlpack_test_cpu PASSED in 28.3s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 27.6s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 28.7s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 34.6s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.2s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 3.1s //tensorflow/c/experimental/grappler:grappler_test PASSED in 24.4s //tensorflow/c/experimental/next_pluggable_device:tensor_pjrt_buffer_util_test PASSED in 5.9s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 0.6s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.5s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.4s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 15.9s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 10.2s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 18.3s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 11.6s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 12.9s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 14.9s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 16.0s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 27.7s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.7s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.6s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 0.5s //tensorflow/c/kernels:summary_op_test PASSED in 0.5s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.6s //tensorflow/cc:cc_op_gen_test PASSED in 0.3s //tensorflow/cc:client_client_session_test PASSED in 4.1s //tensorflow/cc:coordinator_test PASSED in 5.0s //tensorflow/cc:framework_cc_ops_test PASSED in 3.5s //tensorflow/cc:framework_gradient_checker_test PASSED in 3.2s //tensorflow/cc:framework_gradients_test PASSED in 4.9s //tensorflow/cc:framework_scope_test PASSED in 0.6s //tensorflow/cc:framework_while_gradients_test PASSED in 3.1s //tensorflow/cc:gradients_array_grad_test PASSED in 10.5s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 2.1s //tensorflow/cc:gradients_functional_grad_test PASSED in 2.2s //tensorflow/cc:gradients_image_grad_test PASSED in 10.9s //tensorflow/cc:gradients_linalg_grad_test PASSED in 3.2s //tensorflow/cc:gradients_manip_grad_test PASSED in 3.0s //tensorflow/cc:gradients_math_grad_test PASSED in 9.0s //tensorflow/cc:gradients_nn_grad_test PASSED in 5.0s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.2s //tensorflow/cc:ops_const_op_test PASSED in 0.7s //tensorflow/cc:ops_while_loop_test PASSED in 2.0s //tensorflow/cc:queue_runner_test PASSED in 12.6s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.7s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 29.2s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.1s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 26.4s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.7s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 28.9s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 27.8s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.5s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.7s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.6s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.5s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.1s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.2s //tensorflow/cc/saved_model:fingerprinting_chunked_test PASSED in 0.2s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.2s //tensorflow/cc/saved_model:fingerprinting_utils_test PASSED in 0.2s //tensorflow/cc/saved_model:metrics_test PASSED in 0.1s //tensorflow/cc/saved_model:reader_test PASSED in 0.1s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 7.1s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 7.5s //tensorflow/cc/saved_model:util_test PASSED in 0.1s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 33.9s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 2.2s //tensorflow/compiler/aot:codegen_test PASSED in 27.8s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 16.7s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 8.0s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 4.7s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 4.7s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 21.9s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 4.4s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 17.3s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 23.4s //tensorflow/compiler/jit:device_util_test PASSED in 5.8s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 0.8s //tensorflow/compiler/jit:node_matchers_test PASSED in 1.3s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 8.4s //tensorflow/compiler/jit:shape_inference_test PASSED in 0.4s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 21.0s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 9.4s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 5.6s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 8.6s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 22.5s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 24.6s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 16.3s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 15.1s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite:offset_buffer_test PASSED in 0.5s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 13.4s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.1s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 1.8s //tensorflow/compiler/mlir/lite/debug:debug_test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 1.3s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 6.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 0.3s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.1s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:call_xla_module_to_stablehlo.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:compose-uniform-quantized-type.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 4.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 4.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 18.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize_hlo.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tfl_legalize_hlo.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tfl_legalize_hlo_custom_call.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfold_splat_constant_pass.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:uniform-quantized-stablehlo-to-tfl.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 5.1s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:legalize-tensorlist.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 3.6s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 4.0s //tensorflow/compiler/mlir/lite/tests:optimize_batch_matmul.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 6.1s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:reduce-type-precision.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 15.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants_offset.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op_offset.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:mix_tflite_stablehlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_const.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_custom_call.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:variable.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_tensorlist_reserve.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:deduplicate_const.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 3.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 3.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:u16_quant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 5.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variable.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/common:lift_as_function_call_test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test PASSED in 15.9s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_types_test PASSED in 17.2s //tensorflow/compiler/mlir/quantization/stablehlo:math_utils_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo:stablehlo_type_utils_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/stablehlo:tf_type_utils_test PASSED in 23.6s //tensorflow/compiler/mlir/quantization/stablehlo:uniform_quantized_types_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo/cc:export_test PASSED in 10.1s //tensorflow/compiler/mlir/quantization/stablehlo/cc:graph_def_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo/cc:io_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/stablehlo/cc:precalibration_test PASSED in 14.3s //tensorflow/compiler/mlir/quantization/stablehlo/cc/calibration:assign_ids_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo/python:quantize_model_test PASSED in 120.7s //tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 2.2s //tensorflow/compiler/mlir/quantization/stablehlo/tests:stablehlo_op_quant_spec_test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibration_algorithm_test PASSED in 36.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibration_statistics_collector_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 25.0s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:constant_fold_test PASSED in 3.9s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 5.1s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/ops:tf_op_quant_spec_test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/ops:tf_quantize_op_test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 84.9s //tensorflow/compiler/mlir/quantization/tensorflow/python:py_function_lib_py_test PASSED in 22.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 21.2s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 32.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:add_dump_tensor_op.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:add_quantization_unit_loc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_xla_op_to_tf_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_hashtable_ops_as_args.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla_selective_quantization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_duplicate_resource_ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:propagate_quantize_type.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composit_functions_debugging.mlir.test PASSED in 7.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 4.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_weights.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 12.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_uniform_attribute_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 37.7s //tensorflow/compiler/mlir/stablehlo:stablehlo_test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 5.7s //tensorflow/compiler/mlir/tensorflow:call_graph_util_test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.1s //tensorflow/compiler/mlir/tensorflow:data_dumper_logger_config_test PASSED in 5.7s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 13.3s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:xla_rewrite_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:colocate_tpu_copy_with_dynamic_shape.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:embedding_program_key.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:embedding_sequencing.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:move_tpu_compile_to_front.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:print.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 3.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_asset_sinking.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tfrt_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:verify_for_export.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/tensorflow/tests:xla_outline_entry_functions.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite_v2.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:xla_sharding_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow/tests:xla_validate_iputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:convert_mhlo_quant_to_int.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 6.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 3.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 3.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graph:convert_tensor.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 16.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/transforms:verify_no_outside_compilation_markers_pass_test PASSED in 16.0s //tensorflow/compiler/mlir/tensorflow/transforms/host_runtime:lower_cluster_to_runtime_ops_test PASSED in 17.3s //tensorflow/compiler/mlir/tensorflow/transforms/host_runtime:tpu_metadata_utils_test PASSED in 12.3s //tensorflow/compiler/mlir/tensorflow/translate:tf_mlir_translate_registration_test PASSED in 17.5s //tensorflow/compiler/mlir/tf2xla/api/v1:cluster_tf_test PASSED in 19.8s //tensorflow/compiler/mlir/tf2xla/api/v1:compile_mlir_util_test PASSED in 5.2s //tensorflow/compiler/mlir/tf2xla/api/v1:compile_tf_graph_test PASSED in 0.3s //tensorflow/compiler/mlir/tf2xla/api/v1:tf_dialect_to_executor_test PASSED in 14.3s //tensorflow/compiler/mlir/tf2xla/api/v2:cluster_tf_test PASSED in 23.0s //tensorflow/compiler/mlir/tf2xla/api/v2:legalize_tf_test PASSED in 20.9s //tensorflow/compiler/mlir/tf2xla/api/v2:tf_dialect_to_executor_test PASSED in 14.3s //tensorflow/compiler/mlir/tf2xla/internal:clustering_bridge_passes_test PASSED in 6.2s //tensorflow/compiler/mlir/tf2xla/internal:compilation_timer_test PASSED in 0.2s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_mlir_test PASSED in 23.3s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_to_hlo_test PASSED in 21.9s //tensorflow/compiler/mlir/tf2xla/internal:logging_hooks_test PASSED in 14.9s //tensorflow/compiler/mlir/tf2xla/internal:mlir_pass_instrumentation_test PASSED in 6.1s //tensorflow/compiler/mlir/tf2xla/internal:test_matchers_test PASSED in 4.2s //tensorflow/compiler/mlir/tf2xla/internal/inference:inference_metrics_pass_test PASSED in 14.2s //tensorflow/compiler/mlir/tf2xla/internal/passes:tpu_cluster_formation_test PASSED in 19.5s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_clustering_pass_test PASSED in 14.0s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_clustering_pass_test.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_input_dialect_to_executor_pass_test.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/internal/utils:dialect_detection_utils_test PASSED in 0.4s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-quant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 14.5s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/transforms:legalization_op_config_test PASSED in 27.6s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 15.2s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 17.1s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 0.4s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 3.1s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 16.4s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 9.9s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 25.9s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 5.7s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 24.8s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 23.8s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 23.2s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 23.7s //tensorflow/compiler/mlir/tfrt/tests:batch_function_fallback_resource_variable_as_captured_tensor.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tfrt/tests:batch_function_lowering.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:convert_ref_variables.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests:cross_device_transfer.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tfrt/tests:deduplicate_if_results.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:fuse_tpu_compile_and_execute_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops_mlrt.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests:optimize.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tfrt/tests:remove_device_attribute.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests:runtime_lowering_gpu.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:runtime_lowering_tpu.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests:sink_in_invariant_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_fallback.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_lowering.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tfrt/tests:xla_rewrite.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/analysis:cost_analysis.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/analysis:tensor_array_side_effect_analysis.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/analysis:update_op_cost_in_tfrt_mlir_test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/ifrt:rewrite_cluster_to_ifrt_call.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/ir:fallback_opt.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/ir:tfrt_fallback_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tfrt/tests/mlrt:assign_op_key.mlir.test PASSED in 4.0s //tensorflow/compiler/mlir/tfrt/tests/mlrt:async_while.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tfrt/tests/mlrt:fuse_mlrt_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:inline.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:parallelization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tf_to_mlrt.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tpu_conversions.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/mlrt:while_to_map_fn.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:basic.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate_failed.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:const_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:control_flow.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:decompose_resource_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:derived_attrs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:device_conversion.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:errors.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_canonicalization.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_inline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes_multiple_callers.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_use_fallback_tensor.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:insert_fallback_tensor_copy.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:merge_tf_if_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:optimize_tf_control_flow_side_effect.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:remove_tf_if_const_args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:reorder_assert.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:side_effects.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline_refvar.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:whileop.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/translate/mlrt:mlir_to_bytecode_test PASSED in 0.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:func_to_jit_invocations.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 11.8s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 6.0s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-stateful.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 2.6s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 23.1s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 19.3s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 13.7s //tensorflow/compiler/tests:adam_test_cpu PASSED in 32.8s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 9.9s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 15.9s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 18.1s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 10.8s //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 16.9s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 14.0s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 22.5s //tensorflow/compiler/tests:case_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 10.1s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 31.1s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 13.1s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 37.6s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 21.6s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 9.8s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 12.1s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 19.5s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 13.4s //tensorflow/compiler/tests:cond_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:const_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 21.0s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 21.5s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 16.3s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 13.3s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 14.9s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 10.1s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 11.7s //tensorflow/compiler/tests:eager_test_cpu PASSED in 24.2s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 12.8s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 14.6s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 32.3s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 32.3s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 23.6s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 17.2s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 11.6s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 12.2s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 15.1s //tensorflow/compiler/tests:function_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 13.1s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 13.2s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 11.8s //tensorflow/compiler/tests:gather_test_cpu PASSED in 74.5s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 55.2s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 12.8s //tensorflow/compiler/tests:jit_test_cpu PASSED in 67.3s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 13.9s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 18.6s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 12.7s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 43.0s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 18.4s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 40.3s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 20.3s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 39.0s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 17.0s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 14.6s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 14.0s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 12.4s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 18.9s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 13.5s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 10.7s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 11.6s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 11.5s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 17.0s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 16.1s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 17.6s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 13.3s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 12.1s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 11.6s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 14.7s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 17.7s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 10.8s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 13.2s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 35.2s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 23.5s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 45.0s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 12.0s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 12.1s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 25.0s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 50.0s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 20.6s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 20.2s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 19.1s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 29.0s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 12.7s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 16.9s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 17.3s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 12.4s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 15.7s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 23.6s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 15.3s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 15.2s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 13.3s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 26.6s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 24.5s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 16.5s //tensorflow/compiler/tests:while_test_cpu PASSED in 27.1s //tensorflow/compiler/tests:xla_call_module_no_platform_check_test_cpu PASSED in 12.5s //tensorflow/compiler/tests:xla_call_module_no_shape_assertions_check_test_cpu PASSED in 16.6s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 15.5s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 12.0s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 27.7s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 27.3s //tensorflow/compiler/tests:xla_dump_to_test_cpu PASSED in 18.8s //tensorflow/compiler/tests:xla_dump_to_test_cpu_mlir_bridge_test PASSED in 10.3s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 56.5s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 32.3s //tensorflow/compiler/tests:xla_test_test PASSED in 10.1s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 4.6s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.1s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 1.2s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 0.8s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 21.0s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 5.2s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 0.6s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 5.1s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 2.5s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.8s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 7.9s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 17.0s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.7s //tensorflow/compiler/tf2xla:type_util_test PASSED in 1.3s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 16.1s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 14.1s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 4.6s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 1.2s //tensorflow/core:@local_tsl__tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 8.7s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 35.7s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 10.8s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.4s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.5s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.2s //tensorflow/core/api_def:api_test PASSED in 3.0s //tensorflow/core/api_def:update_api_def_test PASSED in 0.2s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 1.1s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 1.7s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 1.1s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 5.7s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 5.4s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 0.7s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.4s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 1.9s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.1s //tensorflow/core/common_runtime:device_mgr_test PASSED in 1.0s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.4s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 0.7s //tensorflow/core/common_runtime:device_set_test PASSED in 0.7s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 3.8s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 3.3s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.2s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 0.8s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.9s //tensorflow/core/common_runtime:executor_test PASSED in 5.2s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 0.7s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 0.7s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 0.7s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.0s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 2.7s //tensorflow/core/common_runtime:graph_runner_test PASSED in 0.8s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 3.4s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 1.0s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.6s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 0.5s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 0.8s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 3.1s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 2.3s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 3.4s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 3.0s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.8s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.1s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.6s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.1s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.1s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 0.8s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 6.5s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 1.2s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 0.8s //tensorflow/core/common_runtime:pending_counts_test PASSED in 1.0s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 3.2s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:placer_test PASSED in 0.7s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 1.4s //tensorflow/core/common_runtime:process_util_test PASSED in 0.1s //tensorflow/core/common_runtime:quantize_training_test PASSED in 2.1s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.1s //tensorflow/core/common_runtime:replicate_constants_pass_test PASSED in 0.7s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.7s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.3s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.3s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 2.4s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 6.9s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 4.5s //tensorflow/core/common_runtime:session_test PASSED in 0.8s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 1.3s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 0.9s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 0.8s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.9s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 25.7s //tensorflow/core/common_runtime/eager:context_test PASSED in 14.1s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 10.1s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 11.3s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 1.2s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 13.1s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 11.8s //tensorflow/core/common_runtime/eager:execute_test PASSED in 27.3s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 1.4s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 13.1s //tensorflow/core/common_runtime/eager:placement_test PASSED in 9.8s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 15.4s //tensorflow/core/common_runtime/eager:summary_optimizer_test PASSED in 0.1s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 12.6s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 11.5s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.4s //tensorflow/core/common_runtime/gpu:gpu_serving_device_selector_test PASSED in 0.5s //tensorflow/core/common_runtime/next_pluggable_device:c_plugin_coordination_service_agent_test PASSED in 3.4s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 26.1s //tensorflow/core/common_runtime/next_pluggable_device/c:tf_rendezvous_c_api_test PASSED in 0.9s //tensorflow/core/config:flags_py_test PASSED in 15.5s //tensorflow/core/config:flags_test PASSED in 0.2s //tensorflow/core/data:compression_utils_test PASSED in 1.7s //tensorflow/core/data:dataset_utils_test PASSED in 0.7s //tensorflow/core/data:hash_utils_test PASSED in 0.8s //tensorflow/core/data:metric_utils_test PASSED in 5.8s //tensorflow/core/data:name_utils_test PASSED in 0.2s //tensorflow/core/data:rewrite_utils_test PASSED in 0.6s //tensorflow/core/data:serialization_utils_test PASSED in 0.8s //tensorflow/core/data:snapshot_utils_test PASSED in 0.5s //tensorflow/core/data:split_utils_test PASSED in 0.5s //tensorflow/core/data:standalone_save_restore_test PASSED in 2.2s //tensorflow/core/data:standalone_test PASSED in 5.9s //tensorflow/core/data:tfdataz_metrics_test PASSED in 1.8s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.7s //tensorflow/core/data:utils_test PASSED in 0.2s //tensorflow/core/data/service:auto_scaler_test PASSED in 0.2s //tensorflow/core/data/service:common_test PASSED in 0.1s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.6s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.4s //tensorflow/core/data/service:data_service_test PASSED in 19.0s //tensorflow/core/data/service:data_transfer_test PASSED in 0.4s //tensorflow/core/data/service:dataset_store_test PASSED in 1.1s //tensorflow/core/data/service:dispatcher_client_test PASSED in 4.9s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.5s //tensorflow/core/data/service:graph_rewriters_test PASSED in 1.3s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 3.0s //tensorflow/core/data/service:grpc_util_test PASSED in 0.6s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 2.5s //tensorflow/core/data/service:journal_test PASSED in 0.4s //tensorflow/core/data/service:logging_utils_test PASSED in 0.2s //tensorflow/core/data/service:split_provider_test PASSED in 2.9s //tensorflow/core/data/service:task_runner_test PASSED in 4.3s //tensorflow/core/data/service:test_util_test PASSED in 2.5s //tensorflow/core/data/service:url_test PASSED in 0.1s //tensorflow/core/data/service:utils_test PASSED in 0.6s //tensorflow/core/data/service:validate_utils_test PASSED in 0.3s //tensorflow/core/data/service:worker_client_test PASSED in 2.9s //tensorflow/core/data/service:worker_impl_test PASSED in 3.2s //tensorflow/core/data/service/client:data_service_client_test PASSED in 6.3s //tensorflow/core/data/service/client:utils_test PASSED in 2.4s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.7s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 18.3s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.7s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.1s //tensorflow/core/data/service/snapshot:snapshot_chunk_provider_test PASSED in 0.3s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 3.5s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 0.8s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 6.0s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 5.2s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.1s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.5s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.1s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 4.4s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 4.7s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.8s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 0.5s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.1s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.3s //tensorflow/core/distributed_runtime:request_id_test PASSED in 1.0s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.5s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.1s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 0.8s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.7s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.9s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 22.5s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 11.3s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 26.0s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 39.6s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 27.1s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 4.2s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 1.0s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 0.6s //tensorflow/core/example:example_parser_configuration_test PASSED in 1.0s //tensorflow/core/example:feature_util_test PASSED in 0.2s //tensorflow/core/framework:allocator_test PASSED in 4.1s //tensorflow/core/framework:attr_value_util_test PASSED in 0.8s //tensorflow/core/framework:batch_util_test PASSED in 1.6s //tensorflow/core/framework:bfloat16_test PASSED in 0.7s //tensorflow/core/framework:common_shape_fns_test PASSED in 0.9s //tensorflow/core/framework:dataset_test PASSED in 1.6s //tensorflow/core/framework:device_base_test PASSED in 0.7s //tensorflow/core/framework:disable_jit_test PASSED in 0.9s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.1s //tensorflow/core/framework:framework_op_segment_test PASSED in 0.8s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.2s //tensorflow/core/framework:framework_run_handler_test PASSED in 2.8s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.2s //tensorflow/core/framework:full_type_inference_util_test PASSED in 0.8s //tensorflow/core/framework:full_type_util_test PASSED in 0.8s //tensorflow/core/framework:function_test PASSED in 1.1s //tensorflow/core/framework:graph_def_util_test PASSED in 2.3s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 0.8s //tensorflow/core/framework:kernel_def_builder_test PASSED in 1.1s //tensorflow/core/framework:kernel_def_util_test PASSED in 1.0s //tensorflow/core/framework:memory_types_test PASSED in 0.8s //tensorflow/core/framework:model_test PASSED in 0.8s //tensorflow/core/framework:node_def_builder_test PASSED in 0.8s //tensorflow/core/framework:node_def_util_test PASSED in 0.9s //tensorflow/core/framework:node_properties_test PASSED in 2.0s //tensorflow/core/framework:op_compatibility_test PASSED in 1.1s //tensorflow/core/framework:op_def_builder_test PASSED in 0.7s //tensorflow/core/framework:op_def_util_test PASSED in 0.8s //tensorflow/core/framework:op_kernel_test PASSED in 0.8s //tensorflow/core/framework:op_registration_test PASSED in 0.8s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 3.2s //tensorflow/core/framework:rendezvous_test PASSED in 3.1s //tensorflow/core/framework:resource_handle_test PASSED in 0.1s //tensorflow/core/framework:resource_mgr_test PASSED in 1.7s //tensorflow/core/framework:resource_op_kernel_test PASSED in 0.8s //tensorflow/core/framework:shape_inference_test PASSED in 1.2s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 1.4s //tensorflow/core/framework:tensor_matcher_test PASSED in 1.4s //tensorflow/core/framework:tensor_shape_test PASSED in 7.1s //tensorflow/core/framework:tensor_slice_test PASSED in 1.1s //tensorflow/core/framework:tensor_test PASSED in 41.1s //tensorflow/core/framework:tensor_testutil_test PASSED in 0.8s //tensorflow/core/framework:tensor_util_test PASSED in 1.2s //tensorflow/core/framework:tracking_allocator_test PASSED in 1.7s //tensorflow/core/framework:types_test PASSED in 1.3s //tensorflow/core/framework:variant_op_registry_test PASSED in 22.8s //tensorflow/core/framework:variant_test PASSED in 8.7s //tensorflow/core/framework/registration:registration_test PASSED in 0.8s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 29.0s //tensorflow/core/function/capture:capture_container_test PASSED in 12.4s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 41.8s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 54.5s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 8.9s //tensorflow/core/function/polymorphism:function_type_test PASSED in 9.8s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 8.7s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 42.1s //tensorflow/core/function/trace_type:custom_nest_trace_type_test PASSED in 15.7s //tensorflow/core/function/trace_type:default_types_test PASSED in 13.3s //tensorflow/core/function/trace_type:serialization_test PASSED in 8.7s //tensorflow/core/function/trace_type:trace_type_test PASSED in 11.6s //tensorflow/core/graph:algorithm_test PASSED in 0.8s //tensorflow/core/graph:collective_order_test PASSED in 0.6s //tensorflow/core/graph:control_flow_test PASSED in 4.6s //tensorflow/core/graph:costmodel_test PASSED in 0.7s //tensorflow/core/graph:edgeset_test PASSED in 1.3s //tensorflow/core/graph:graph_debug_info_builder_test PASSED in 0.7s //tensorflow/core/graph:graph_def_builder_test PASSED in 1.5s //tensorflow/core/graph:graph_partition_test PASSED in 1.0s //tensorflow/core/graph:graph_test PASSED in 1.2s //tensorflow/core/graph:node_builder_test PASSED in 1.0s //tensorflow/core/graph:optimizer_cse_test PASSED in 0.8s //tensorflow/core/graph:subgraph_test PASSED in 1.9s //tensorflow/core/graph:tensor_id_test PASSED in 1.4s //tensorflow/core/graph:validate_test PASSED in 0.8s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.3s //tensorflow/core/graph/regularization:util_test PASSED in 0.1s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.5s //tensorflow/core/grappler:graph_view_test PASSED in 1.4s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 2.2s //tensorflow/core/grappler:grappler_item_test PASSED in 2.2s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 1.8s //tensorflow/core/grappler:utils_test PASSED in 3.4s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 28.9s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 1.5s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 2.0s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.5s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.8s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 3.2s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.1s //tensorflow/core/grappler/costs:utils_test PASSED in 1.5s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.5s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.7s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 2.1s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 1.9s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 1.6s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 3.1s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 2.8s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 5.0s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 6.0s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.7s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 4.6s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 1.9s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.4s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.5s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 2.2s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 1.8s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 8.6s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 2.9s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 4.1s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 9.8s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:inject_io_prefetch_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:remove_compression_map_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 1.4s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 1.6s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.3s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 2.9s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.5s //tensorflow/core/grappler/utils:frame_test PASSED in 0.2s //tensorflow/core/grappler/utils:functions_test PASSED in 1.5s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.4s //tensorflow/core/grappler/utils:graph_view_test PASSED in 2.0s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 7.1s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.5s //tensorflow/core/grappler/utils:scc_test PASSED in 1.9s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.1s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.4s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.1s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.7s //tensorflow/core/grappler/utils:traversal_test PASSED in 1.1s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.8s //tensorflow/core/ir:interfaces_test PASSED in 0.7s //tensorflow/core/ir:ops_test PASSED in 0.3s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.4s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.5s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.2s //tensorflow/core/ir:utility_test PASSED in 0.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 2.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 13.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 2.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 2.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 2.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 1.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 16.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 15.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 1.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 2.0s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.8s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.7s //tensorflow/core/ir/types:dialect_test PASSED in 0.1s //tensorflow/core/kernels:as_string_op_test PASSED in 1.2s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.7s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 5.0s //tensorflow/core/kernels:batch_kernels_test PASSED in 43.5s //tensorflow/core/kernels:bias_op_test PASSED in 0.5s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 1.0s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.6s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.5s //tensorflow/core/kernels:concat_op_test PASSED in 0.8s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:control_flow_ops_test PASSED in 7.9s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.4s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.4s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 6.3s //tensorflow/core/kernels:count_ops_test PASSED in 0.4s //tensorflow/core/kernels:cross_op_test PASSED in 0.6s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:debug_ops_test PASSED in 0.7s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.3s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.4s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.5s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.6s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.2s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 0.4s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.1s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.9s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.1s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.5s //tensorflow/core/kernels:encode_wav_op_test PASSED in 2.4s //tensorflow/core/kernels:fingerprint_op_test PASSED in 0.5s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 0.5s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.5s //tensorflow/core/kernels:identity_op_test PASSED in 0.8s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 1.0s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.4s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 0.5s //tensorflow/core/kernels:logging_ops_test PASSED in 1.6s //tensorflow/core/kernels:lookup_ops_test PASSED in 1.3s //tensorflow/core/kernels:loss_test PASSED in 0.2s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 8.1s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 0.9s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.5s //tensorflow/core/kernels:mfcc_test PASSED in 0.2s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.4s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.8s //tensorflow/core/kernels:ops_util_test PASSED in 0.1s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:parse_tensor_test PASSED in 0.8s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.6s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.6s //tensorflow/core/kernels:quantize_op_test PASSED in 0.5s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_add_op_test PASSED in 1.9s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 1.5s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.5s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.0s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.4s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 1.2s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 1.8s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 1.6s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 0.9s //tensorflow/core/kernels:ragged_range_op_test PASSED in 0.5s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.8s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 0.8s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 3.3s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 1.9s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.3s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:range_sampler_test PASSED in 0.2s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.4s //tensorflow/core/kernels:requantization_range_op_test PASSED in 0.9s //tensorflow/core/kernels:requantize_op_test PASSED in 0.8s //tensorflow/core/kernels:resource_ops_test PASSED in 0.7s //tensorflow/core/kernels:restore_op_test PASSED in 0.8s //tensorflow/core/kernels:restore_v2_op_test PASSED in 1.2s //tensorflow/core/kernels:reverse_op_test PASSED in 0.7s //tensorflow/core/kernels:roll_op_test PASSED in 16.3s //tensorflow/core/kernels:save_op_test PASSED in 0.6s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.5s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:scatter_op_test PASSED in 0.6s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 10.2s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.6s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 0.7s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.9s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.7s //tensorflow/core/kernels:shape_ops_test PASSED in 0.5s //tensorflow/core/kernels:slice_op_test PASSED in 0.5s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 2.1s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.6s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.7s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 4.2s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:sparse_utils_test PASSED in 2.6s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 2.2s //tensorflow/core/kernels:spectrogram_test PASSED in 0.4s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.5s //tensorflow/core/kernels:string_format_op_test PASSED in 0.8s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 1.1s //tensorflow/core/kernels:string_split_op_test PASSED in 1.6s //tensorflow/core/kernels:substr_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.7s //tensorflow/core/kernels:summary_image_op_test PASSED in 1.4s //tensorflow/core/kernels:summary_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.5s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.6s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.1s //tensorflow/core/kernels:tensor_map_test PASSED in 0.7s //tensorflow/core/kernels:training_ops_test PASSED in 0.4s //tensorflow/core/kernels:transpose_util_test PASSED in 0.4s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 2.1s //tensorflow/core/kernels:unique_op_test PASSED in 0.7s //tensorflow/core/kernels:variable_ops_test PASSED in 2.3s //tensorflow/core/kernels:while_op_test PASSED in 1.0s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.7s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.5s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 31.9s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.3s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 1.8s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 2.6s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 3.4s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.3s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 1.1s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:get_options_op_test PASSED in 1.7s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 0.7s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.6s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 5.9s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 2.2s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.4s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 3.2s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 2.8s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 2.1s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.5s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 0.8s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 1.8s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.5s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 0.8s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.6s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 0.6s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.0s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 2.4s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 2.2s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 1.7s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 1.6s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.2s //tensorflow/core/kernels/sparse:kernels_test PASSED in 0.4s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.4s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.4s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 2.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 1.8s //tensorflow/core/lib/db:sqlite_test PASSED in 0.2s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 4.0s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.7s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 0.5s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.4s //tensorflow/core/ops:ops_math_grad_test PASSED in 6.2s //tensorflow/core/ops:ops_tests PASSED in 0.6s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.4s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.2s //tensorflow/core/platform:env_test PASSED in 2.5s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 1.0s //tensorflow/core/platform:platform_strings_test PASSED in 0.2s //tensorflow/core/platform:ram_file_system_test PASSED in 12.3s //tensorflow/core/platform:resource_loader_test PASSED in 0.3s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.3s //tensorflow/core/platform:vmodule_test PASSED in 0.2s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.1s //tensorflow/core/profiler/convert:dcn_analysis_test PASSED in 0.1s //tensorflow/core/profiler/convert:dcn_utils_test PASSED in 0.8s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_combiner_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:repository_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_dcn_collective_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.5s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.4s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 1.6s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 0.8s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 0.7s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 0.6s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 1.3s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 0.5s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.1s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.4s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.2s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.3s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.1s //tensorflow/core/runtime_fallback/util:type_util_test PASSED in 0.5s //tensorflow/core/summary:schema_test PASSED in 0.1s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.3s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.5s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 6.3s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 6.8s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 6.4s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.6s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.5s //tensorflow/core/tfrt/graph_executor:config_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/attribute:attribute_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/bytecode:bytecode_test PASSED in 0.4s //tensorflow/core/tfrt/mlrt/bytecode:executable_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:function_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:kernel_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:span_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/interpreter:context_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:future_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/interpreter:interpreter_test PASSED in 0.6s //tensorflow/core/tfrt/mlrt/interpreter:register_span_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/interpreter:value_test PASSED in 0.1s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_concurrent_work_queue_test PASSED in 1.4s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_test PASSED in 1.7s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_util_test PASSED in 0.1s //tensorflow/core/tfrt/runtime:tf_threadpool_concurrent_work_queue_test PASSED in 1.0s //tensorflow/core/tfrt/runtime:work_queue_interface_test PASSED in 0.1s //tensorflow/core/tfrt/utils:graph_partition_test PASSED in 2.2s //tensorflow/core/transforms:eval_utils_test PASSED in 1.4s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.2s //tensorflow/core/util:bcast_test PASSED in 1.2s //tensorflow/core/util:command_line_flags_test PASSED in 0.7s //tensorflow/core/util:debug_data_dumper_test PASSED in 0.7s //tensorflow/core/util:debug_events_writer_test PASSED in 0.4s //tensorflow/core/util:dump_graph_test PASSED in 2.1s //tensorflow/core/util:equal_graph_def_test PASSED in 0.7s //tensorflow/core/util:events_writer_test PASSED in 2.8s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.5s //tensorflow/core/util:example_proto_helper_test PASSED in 0.9s //tensorflow/core/util:exec_on_stall_test PASSED in 2.3s //tensorflow/core/util:fake_clock_env_test PASSED in 2.4s //tensorflow/core/util:incremental_barrier_test PASSED in 0.4s //tensorflow/core/util:matmul_bcast_test PASSED in 0.8s //tensorflow/core/util:memmapped_file_system_test PASSED in 0.7s //tensorflow/core/util:mkl_heuristics_test PASSED in 0.1s //tensorflow/core/util:overflow_test PASSED in 0.1s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 2.3s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.7s //tensorflow/core/util:reffed_status_callback_test PASSED in 0.8s //tensorflow/core/util:reporter_test PASSED in 0.9s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.7s //tensorflow/core/util:semver_test PASSED in 0.7s //tensorflow/core/util:stat_summarizer_test PASSED in 1.3s //tensorflow/core/util:strided_slice_op_test PASSED in 1.5s //tensorflow/core/util:tensor_format_test PASSED in 0.7s //tensorflow/core/util:tensor_slice_reader_test PASSED in 2.1s //tensorflow/core/util:tensor_slice_set_test PASSED in 0.7s //tensorflow/core/util:tensor_slice_util_test PASSED in 0.7s //tensorflow/core/util:tensor_slice_writer_test PASSED in 2.9s //tensorflow/core/util:work_sharder_test PASSED in 1.3s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.1s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.5s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.5s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.2s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.1s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 44.3s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.1s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:decompose_controlflow.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 1.4s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_collective_type_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 3.0s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 10.1s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 2.6s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 17.1s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 2.1s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 1.9s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 1.4s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 2.9s //tensorflow/dtensor/mlir/tests:spmd_fft.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 2.1s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 1.9s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 13.2s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 0.7s //tensorflow/dtensor/python/tests:api_test PASSED in 30.0s //tensorflow/dtensor/python/tests:array_ops_test_cpu PASSED in 23.2s //tensorflow/dtensor/python/tests:cache_test_cpu PASSED in 19.0s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 21.8s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 22.1s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 23.8s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 39.6s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 18.5s //tensorflow/dtensor/python/tests:mesh_util_test_cpu PASSED in 18.0s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 15.5s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 13.4s //tensorflow/dtensor/python/tests:variable_test_cpu PASSED in 18.4s //tensorflow/dtensor/tests:dtensor_operation_test PASSED in 33.6s //tensorflow/dtensor/tests:executable_manager_test PASSED in 32.6s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.2s //tensorflow/dtensor/tests:slice_util_test PASSED in 0.2s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 6.1s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.1s //tensorflow/examples/adding_an_op:fact_test PASSED in 22.1s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 28.3s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 51.8s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 21.9s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 47.1s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 40.5s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 45.4s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 37.4s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 49.3s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 27.0s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 3.3s //tensorflow/examples/speech_commands:models_test PASSED in 23.0s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.1s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.0s //tensorflow/js:ts_op_gen_test PASSED in 0.3s //tensorflow/python/autograph/converters:asserts_test PASSED in 10.2s //tensorflow/python/autograph/converters:break_statements_test PASSED in 9.3s //tensorflow/python/autograph/converters:call_trees_test PASSED in 10.5s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 15.8s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 11.3s //tensorflow/python/autograph/converters:control_flow_test PASSED in 16.4s //tensorflow/python/autograph/converters:directives_test PASSED in 9.7s //tensorflow/python/autograph/converters:functions_test PASSED in 13.0s //tensorflow/python/autograph/converters:lists_test PASSED in 11.2s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 14.9s //tensorflow/python/autograph/converters:return_statements_test PASSED in 18.5s //tensorflow/python/autograph/converters:slices_test PASSED in 9.2s //tensorflow/python/autograph/converters:variables_test PASSED in 10.0s //tensorflow/python/autograph/core:converter_test PASSED in 11.3s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 14.0s //tensorflow/python/autograph/impl:api_test PASSED in 27.1s //tensorflow/python/autograph/impl:conversion_test PASSED in 12.2s //tensorflow/python/autograph/lang:special_functions_test PASSED in 11.9s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 10.0s //tensorflow/python/autograph/operators:control_flow_test PASSED in 25.8s //tensorflow/python/autograph/operators:data_structures_test PASSED in 13.0s //tensorflow/python/autograph/operators:exceptions_test PASSED in 9.2s //tensorflow/python/autograph/operators:logical_test PASSED in 12.7s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 18.9s //tensorflow/python/autograph/operators:slices_test PASSED in 9.8s //tensorflow/python/autograph/operators:variables_test PASSED in 11.4s //tensorflow/python/autograph/pyct:anno_test PASSED in 10.0s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 18.1s //tensorflow/python/autograph/pyct:cache_test PASSED in 13.1s //tensorflow/python/autograph/pyct:cfg_test PASSED in 11.4s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 9.1s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 10.6s //tensorflow/python/autograph/pyct:loader_test PASSED in 10.7s //tensorflow/python/autograph/pyct:naming_test PASSED in 18.4s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 12.8s //tensorflow/python/autograph/pyct:parser_test PASSED in 10.6s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 9.1s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 11.4s //tensorflow/python/autograph/pyct:templates_test PASSED in 13.3s //tensorflow/python/autograph/pyct:transformer_test PASSED in 11.6s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 10.1s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 12.7s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 9.6s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 10.2s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 11.7s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 17.0s //tensorflow/python/autograph/tests:assertion_test PASSED in 25.7s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 24.1s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 33.5s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 28.1s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 26.1s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 34.0s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 27.8s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 22.2s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 36.1s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 40.1s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 55.7s //tensorflow/python/autograph/tests:datasets_test PASSED in 27.6s //tensorflow/python/autograph/tests:early_return_test PASSED in 62.6s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 24.3s //tensorflow/python/autograph/tests:generator_test PASSED in 30.6s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 28.3s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 101.7s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 25.5s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 29.6s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 64.2s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 72.3s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 29.7s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 40.5s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 71.1s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 23.8s //tensorflow/python/autograph/utils:context_managers_test PASSED in 10.1s //tensorflow/python/autograph/utils:misc_test PASSED in 10.0s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 11.7s //tensorflow/python/autograph/utils:tensors_test PASSED in 9.6s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 20.9s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 16.1s //tensorflow/python/checkpoint:checkpoint_test PASSED in 29.3s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 11.0s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 17.7s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 18.7s //tensorflow/python/checkpoint:restore_test PASSED in 12.5s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 12.8s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 13.7s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 10.5s //tensorflow/python/checkpoint:trackable_view_test PASSED in 10.8s //tensorflow/python/checkpoint/sharding:sharding_policies_test PASSED in 20.1s //tensorflow/python/checkpoint/sharding:sharding_util_test PASSED in 13.5s //tensorflow/python/client:device_lib_test_cpu PASSED in 9.6s //tensorflow/python/client:events_writer_test PASSED in 16.1s //tensorflow/python/client:session_list_devices_test PASSED in 10.9s //tensorflow/python/client:session_partial_run_test PASSED in 31.2s //tensorflow/python/client:timeline_test_cpu PASSED in 12.6s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 9.9s //tensorflow/python/compat:compat_test PASSED in 11.7s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 18.7s //tensorflow/python/compiler/mlir:mlir_test PASSED in 15.4s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 14.2s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 11.3s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 13.1s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 15.5s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 16.8s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 28.1s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 12.8s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 16.7s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 15.7s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 11.0s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 13.1s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 11.1s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 26.7s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 15.1s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 11.5s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 12.1s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 10.7s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 16.7s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 14.2s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 15.4s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 14.1s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 15.0s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 12.0s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 22.2s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 41.9s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 10.4s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 45.2s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 18.4s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 12.1s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 36.8s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 25.2s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 18.9s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 24.3s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 33.3s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 61.0s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 14.7s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 40.3s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 11.1s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 71.6s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 15.5s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 20.5s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 12.9s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 10.9s //tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 17.3s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 18.3s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 24.1s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 23.5s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 10.8s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 12.7s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 11.0s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 11.7s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 49.2s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 72.2s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 16.7s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 35.4s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 18.9s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 22.0s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 193.5s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 15.5s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 16.8s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 40.7s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 10.8s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 29.7s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 20.5s //tensorflow/python/data/kernel_tests:cache_test PASSED in 59.1s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 17.8s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 24.0s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 42.5s //tensorflow/python/data/kernel_tests:counter_test PASSED in 50.3s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 11.7s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 59.4s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 26.5s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 12.9s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 31.6s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 43.9s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 33.1s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 26.7s //tensorflow/python/data/kernel_tests:io_test PASSED in 53.7s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 42.1s //tensorflow/python/data/kernel_tests:len_test PASSED in 28.2s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 13.2s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 13.4s //tensorflow/python/data/kernel_tests:options_test PASSED in 13.2s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 16.2s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 69.8s //tensorflow/python/data/kernel_tests:random_test PASSED in 31.1s //tensorflow/python/data/kernel_tests:range_test PASSED in 59.0s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 41.8s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 31.7s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 73.7s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 24.6s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 66.1s //tensorflow/python/data/util:convert_test PASSED in 13.9s //tensorflow/python/data/util:nest_test PASSED in 11.3s //tensorflow/python/data/util:options_test PASSED in 13.8s //tensorflow/python/data/util:random_seed_test PASSED in 14.4s //tensorflow/python/data/util:sparse_test PASSED in 12.3s //tensorflow/python/data/util:structure_test PASSED in 13.9s //tensorflow/python/data/util:traverse_test PASSED in 10.9s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 10.9s //tensorflow/python/debug/cli:cli_config_test PASSED in 9.6s //tensorflow/python/debug/cli:cli_shared_test PASSED in 11.9s //tensorflow/python/debug/cli:command_parser_test PASSED in 9.5s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 10.1s //tensorflow/python/debug/cli:evaluator_test PASSED in 9.8s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 11.0s //tensorflow/python/debug/cli:readline_ui_test PASSED in 11.3s //tensorflow/python/debug/cli:tensor_format_test PASSED in 11.4s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 32.8s //tensorflow/python/debug/lib:common_test PASSED in 9.9s //tensorflow/python/debug/lib:debug_data_test PASSED in 11.7s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 13.1s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 12.8s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 13.1s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 15.1s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 11.1s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 12.0s //tensorflow/python/debug/lib:debug_utils_test PASSED in 34.2s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 21.8s //tensorflow/python/debug/lib:profiling_test PASSED in 10.2s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 27.5s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 12.1s //tensorflow/python/debug/lib:source_utils_test PASSED in 13.8s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 14.2s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 27.9s //tensorflow/python/debug/wrappers:framework_test PASSED in 11.2s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 17.5s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 16.8s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 17.2s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 21.3s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 14.3s //tensorflow/python/distribute:collective_util_test PASSED in 13.0s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 28.5s //tensorflow/python/distribute:combinations_test_cpu PASSED in 26.2s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 15.0s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 31.4s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 32.8s //tensorflow/python/distribute:device_util_test_cpu PASSED in 15.2s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 21.0s //tensorflow/python/distribute:distribute_lib_test PASSED in 22.9s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 16.7s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 12.4s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 16.7s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 46.7s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 37.2s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 14.3s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 13.9s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 49.3s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 33.7s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 10.9s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 44.5s //tensorflow/python/distribute:multi_worker_util_test PASSED in 21.8s //tensorflow/python/distribute:numpy_dataset_test PASSED in 12.6s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 30.9s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 11.7s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 42.7s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 39.8s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 23.2s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 43.0s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 15.6s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 13.0s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 12.4s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 11.6s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 12.9s //tensorflow/python/distribute:sharded_variable_test PASSED in 25.1s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 10.7s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 47.4s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 11.7s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 37.9s //tensorflow/python/distribute:test_util_test_cpu PASSED in 24.5s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 16.2s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 13.8s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 18.2s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 32.1s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 12.7s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 10.0s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 9.7s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 9.6s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 12.2s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 25.9s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 16.2s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 15.5s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 64.5s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 16.9s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 61.1s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 30.2s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 64.3s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 16.8s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 65.7s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 88.7s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 10.2s //tensorflow/python/eager:backprop_test_cpu PASSED in 157.9s //tensorflow/python/eager:cancellation_test_cpu PASSED in 11.3s //tensorflow/python/eager:context_test_cpu PASSED in 11.1s //tensorflow/python/eager:core_test_cpu PASSED in 22.6s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 49.5s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 9.5s //tensorflow/python/eager:lift_to_graph_test PASSED in 13.1s //tensorflow/python/eager:monitoring_test_cpu PASSED in 11.7s //tensorflow/python/eager:ops_test_cpu PASSED in 9.5s //tensorflow/python/eager:profiler_client_test PASSED in 9.5s //tensorflow/python/eager:profiler_test_cpu PASSED in 11.3s //tensorflow/python/eager:pywrap_tfe_test PASSED in 26.3s //tensorflow/python/eager:record_test PASSED in 10.2s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 12.3s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 10.8s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 229.8s //tensorflow/python/eager:tensor_test_cpu PASSED in 13.9s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 29.2s //tensorflow/python/eager:wrap_function_test PASSED in 17.4s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 10.2s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 12.8s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 28.8s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 10.1s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 12.0s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 12.7s //tensorflow/python/eager/polymorphic_function:concrete_function_test_cpu PASSED in 10.7s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 10.2s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 10.8s //tensorflow/python/eager/polymorphic_function:tracing_compilation_test PASSED in 24.4s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 14.4s //tensorflow/python/feature_column:serialization_test PASSED in 24.5s //tensorflow/python/framework:auto_control_deps_test PASSED in 31.0s //tensorflow/python/framework:c_api_util_test PASSED in 10.7s //tensorflow/python/framework:common_shapes_test PASSED in 10.4s //tensorflow/python/framework:composite_tensor_test PASSED in 11.1s //tensorflow/python/framework:config_test_2gpu PASSED in 14.4s //tensorflow/python/framework:config_test_cpu PASSED in 16.0s //tensorflow/python/framework:constant_op_test PASSED in 12.4s //tensorflow/python/framework:device_spec_test PASSED in 9.6s //tensorflow/python/framework:device_test PASSED in 21.4s //tensorflow/python/framework:dtypes_test PASSED in 24.3s //tensorflow/python/framework:error_interpolation_test PASSED in 16.7s //tensorflow/python/framework:errors_test PASSED in 17.8s //tensorflow/python/framework:extension_type_field_test PASSED in 34.1s //tensorflow/python/framework:extension_type_test PASSED in 45.1s //tensorflow/python/framework:file_system_test PASSED in 10.5s //tensorflow/python/framework:flexible_dtypes_test PASSED in 108.6s //tensorflow/python/framework:function_def_to_graph_test PASSED in 10.5s //tensorflow/python/framework:graph_util_test PASSED in 12.0s //tensorflow/python/framework:immutable_dict_test PASSED in 13.4s //tensorflow/python/framework:importer_test PASSED in 11.6s //tensorflow/python/framework:indexed_slices_test PASSED in 10.6s //tensorflow/python/framework:kernels_test PASSED in 12.4s //tensorflow/python/framework:meta_graph_test PASSED in 15.6s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 11.0s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.4s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 4.7s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 27.5s //tensorflow/python/framework:op_def_library_test PASSED in 22.5s //tensorflow/python/framework:op_def_util_test PASSED in 10.4s //tensorflow/python/framework:ops_enable_eager_test PASSED in 3.3s //tensorflow/python/framework:ops_test PASSED in 24.2s //tensorflow/python/framework:proto_test PASSED in 10.6s //tensorflow/python/framework:py_context_manager_test PASSED in 8.9s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 12.0s //tensorflow/python/framework:python_api_info_test PASSED in 9.8s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 14.1s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 5.5s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 2.1s //tensorflow/python/framework:python_op_gen_test PASSED in 0.5s //tensorflow/python/framework:python_tensor_converter_test PASSED in 10.8s //tensorflow/python/framework:random_seed_test PASSED in 10.6s //tensorflow/python/framework:registry_test PASSED in 10.0s //tensorflow/python/framework:smart_cond_test PASSED in 12.3s //tensorflow/python/framework:sparse_tensor_test PASSED in 10.8s //tensorflow/python/framework:subscribe_test PASSED in 26.6s //tensorflow/python/framework:tensor_shape_test PASSED in 12.1s //tensorflow/python/framework:tensor_test PASSED in 34.3s //tensorflow/python/framework:tensor_util_test PASSED in 11.8s //tensorflow/python/framework:test_combinations_test PASSED in 11.4s //tensorflow/python/framework:test_util_test_cpu PASSED in 19.3s //tensorflow/python/framework:tf2_test PASSED in 13.4s //tensorflow/python/framework:traceable_stack_test PASSED in 10.1s //tensorflow/python/framework:type_spec_test PASSED in 12.4s //tensorflow/python/framework:versions_test PASSED in 9.2s //tensorflow/python/framework:weak_tensor_test PASSED in 12.2s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 19.6s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 9.8s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 20.3s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 11.8s //tensorflow/python/grappler:cost_analyzer_test PASSED in 16.6s //tensorflow/python/grappler:datasets_test PASSED in 15.2s //tensorflow/python/grappler:item_test PASSED in 10.9s //tensorflow/python/grappler:memory_optimizer_test PASSED in 20.5s //tensorflow/python/grappler:model_analyzer_test PASSED in 10.0s //tensorflow/python/grappler:remapper_test_cpu PASSED in 12.1s //tensorflow/python/grappler:tf_optimizer_test PASSED in 8.5s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 20.6s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 50.5s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 9.9s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 23.0s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 13.6s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 35.4s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 17.5s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests:template_test PASSED in 13.3s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 10.9s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 12.5s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 15.8s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 10.3s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 35.3s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 11.0s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 36.4s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 12.1s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 10.1s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 9.9s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 11.8s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 45.4s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 12.8s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 19.4s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 28.3s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 21.3s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 17.5s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 16.9s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 23.0s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 62.7s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 10.0s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 11.5s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 21.7s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 89.0s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 84.8s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 32.9s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 27.9s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 12.9s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 15.9s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 14.4s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 12.9s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 45.9s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 10.2s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 23.4s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 15.0s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 27.9s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 21.6s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 16.2s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 19.6s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 18.1s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 76.1s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 35.9s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 27.3s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 33.0s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 30.4s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 10.3s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 11.9s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 29.1s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 12.9s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 13.2s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 27.1s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 21.2s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 13.9s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 28.0s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 9.6s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 12.7s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 37.5s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 14.9s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 33.5s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 10.5s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 31.3s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 18.8s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 30.4s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 65.5s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 48.9s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 29.1s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 26.5s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 21.2s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 19.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 11.0s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 15.9s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 10.9s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 14.4s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 13.0s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 13.7s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 46.8s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 148.4s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 8.9s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 18.2s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 17.4s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 98.0s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 58.6s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 12.8s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 31.2s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 40.8s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 41.4s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 14.5s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 131.4s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 15.9s //tensorflow/python/kernel_tests/proto:decode_proto_op_test PASSED in 10.8s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 9.2s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 17.1s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 13.2s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 10.0s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 25.1s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 17.0s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 15.1s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 29.9s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 22.6s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 175.3s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 20.3s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 23.7s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 20.1s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 28.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 13.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 12.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 14.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 42.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 64.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 13.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 11.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 19.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 12.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 37.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 33.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 9.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 91.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 9.9s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 9.5s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 15.5s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 9.4s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 15.6s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 10.0s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 9.3s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 9.3s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 18.4s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 13.5s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 14.8s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 26.3s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 10.6s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 21.4s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 12.3s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 9.5s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 8.9s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 36.8s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 14.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 11.9s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 32.2s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 23.6s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 15.7s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 63.5s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 32.9s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 12.8s //tensorflow/python/lib/io:file_io_test PASSED in 32.8s //tensorflow/python/lib/io:tf_record_test PASSED in 11.9s //tensorflow/python/module:module_test PASSED in 13.0s //tensorflow/python/ops:array_grad_test_cpu PASSED in 14.1s //tensorflow/python/ops:array_ops_shape_test PASSED in 9.3s //tensorflow/python/ops:array_ops_test PASSED in 9.4s //tensorflow/python/ops:autograph_ops_test PASSED in 11.0s //tensorflow/python/ops:bincount_ops_test_cpu PASSED in 17.3s //tensorflow/python/ops:bitwise_ops_test_cpu PASSED in 16.1s //tensorflow/python/ops:clip_ops_test PASSED in 10.4s //tensorflow/python/ops:clustering_ops_test PASSED in 32.0s //tensorflow/python/ops:collective_ops_gpu_test_cpu PASSED in 29.0s //tensorflow/python/ops:collective_ops_test PASSED in 17.9s //tensorflow/python/ops:collective_ops_xla_test PASSED in 9.7s //tensorflow/python/ops:compiled_collective_ops_gpu_test_2gpu PASSED in 17.6s //tensorflow/python/ops:compiled_collective_ops_gpu_test_cpu PASSED in 16.8s //tensorflow/python/ops:control_flow_v2_enable_test PASSED in 9.1s //tensorflow/python/ops:control_flow_v2_toggles_test PASSED in 12.9s //tensorflow/python/ops:dequantize_op_test PASSED in 10.1s //tensorflow/python/ops:embedding_ops_test_cpu PASSED in 17.5s //tensorflow/python/ops:factory_ops_test_cpu PASSED in 24.2s //tensorflow/python/ops:functional_ops_test PASSED in 9.8s //tensorflow/python/ops:gradient_checker_v2_test_cpu PASSED in 30.0s //tensorflow/python/ops:gradients_test_cpu PASSED in 27.1s //tensorflow/python/ops:init_ops_test_cpu PASSED in 10.5s //tensorflow/python/ops:init_ops_v2_test_cpu PASSED in 30.2s //tensorflow/python/ops:lookup_ops_async_checkpoint_test PASSED in 14.4s //tensorflow/python/ops:math_grad_test_cpu PASSED in 21.4s //tensorflow/python/ops:math_ops_linspace_test_cpu PASSED in 11.2s //tensorflow/python/ops:math_ops_test_cpu PASSED in 28.3s //tensorflow/python/ops:nn_grad_test_cpu PASSED in 12.5s //tensorflow/python/ops:nn_loss_scaling_utilities_test PASSED in 15.0s //tensorflow/python/ops:nn_test_cpu PASSED in 71.8s //tensorflow/python/ops:nn_xent_test_cpu PASSED in 16.0s //tensorflow/python/ops:op_selector_test PASSED in 10.4s //tensorflow/python/ops:quantized_conv_ops_test PASSED in 10.4s //tensorflow/python/ops:quantized_ops_test PASSED in 11.9s //tensorflow/python/ops:raw_ops_test_cpu PASSED in 9.2s //tensorflow/python/ops:rnn_grad_test_cpu PASSED in 8.6s //tensorflow/python/ops:script_ops_test PASSED in 11.5s //tensorflow/python/ops:sort_ops_test PASSED in 14.5s //tensorflow/python/ops:sparse_bincount_ops_test_cpu PASSED in 17.5s //tensorflow/python/ops:sparse_ops_test PASSED in 17.9s //tensorflow/python/ops:tensor_array_ops_test PASSED in 9.6s //tensorflow/python/ops:variable_spec_test PASSED in 9.6s //tensorflow/python/ops:weak_tensor_array_ops_test PASSED in 11.1s //tensorflow/python/ops:weak_tensor_constant_op_test PASSED in 16.6s //tensorflow/python/ops:weak_tensor_image_ops_test PASSED in 15.0s //tensorflow/python/ops:weak_tensor_math_ops_test PASSED in 45.2s //tensorflow/python/ops:weak_tensor_nn_test_cpu PASSED in 15.3s //tensorflow/python/ops:weak_tensor_np_array_ops_test PASSED in 44.5s //tensorflow/python/ops:weak_tensor_np_math_ops_test PASSED in 16.3s //tensorflow/python/ops:weak_tensor_ops_test PASSED in 91.6s //tensorflow/python/ops/losses:util_test PASSED in 9.9s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 18.0s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 99.3s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 13.7s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 31.7s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 97.3s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 13.2s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 34.1s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 78.9s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 9.9s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 22.8s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 26.8s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 62.7s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 20.4s //tensorflow/python/ops/parallel_for:pfor_test PASSED in 9.0s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 68.0s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 17.5s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 49.3s //tensorflow/python/ops/ragged:ragged_bincount_ops_test_cpu PASSED in 11.8s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 11.7s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 25.8s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 13.9s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 10.6s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 9.6s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 23.0s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 165.2s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 28.7s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 8.9s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 9.7s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 34.0s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 14.2s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 14.5s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 27.8s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 11.9s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 12.9s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 15.8s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 33.2s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 46.3s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 31.4s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 15.0s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 23.7s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 9.2s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 16.6s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 10.2s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 8.7s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 59.3s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 37.9s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 10.0s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 11.2s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 11.5s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 11.6s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 16.7s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 11.5s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 47.5s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 19.0s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 17.3s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 29.2s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 88.5s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 55.7s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 26.4s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 97.0s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 29.4s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 38.8s //tensorflow/python/ops/ragged:row_partition_test PASSED in 33.8s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 9.2s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 27.8s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 75.5s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 84.3s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 15.4s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 60.6s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 13.2s //tensorflow/python/platform:benchmark_test PASSED in 17.3s //tensorflow/python/platform:build_info_test PASSED in 11.6s //tensorflow/python/platform:resource_loader_test PASSED in 7.6s //tensorflow/python/profiler:pprof_profiler_test PASSED in 26.0s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 25.6s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 9.6s //tensorflow/python/profiler:profiler_test_cpu PASSED in 20.4s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 14.5s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 12.4s //tensorflow/python/profiler:tfprof_logger_test PASSED in 12.0s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 10.6s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 11.5s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 41.0s //tensorflow/python/saved_model:fingerprinting_test PASSED in 13.4s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 22.2s //tensorflow/python/saved_model:loader_test PASSED in 14.8s //tensorflow/python/saved_model:method_name_updater_test PASSED in 9.8s //tensorflow/python/saved_model:metrics_test PASSED in 11.4s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 13.9s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 12.1s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 9.0s //tensorflow/python/saved_model:revived_types_test PASSED in 9.2s //tensorflow/python/saved_model:save_context_test PASSED in 9.3s //tensorflow/python/saved_model:save_test PASSED in 45.1s //tensorflow/python/saved_model:saved_model_test PASSED in 41.1s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 10.1s //tensorflow/python/saved_model:simple_save_test PASSED in 10.3s //tensorflow/python/saved_model:tracing_utils_test PASSED in 10.9s //tensorflow/python/saved_model:utils_test PASSED in 11.4s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 11.5s //tensorflow/python/saved_model/model_utils:export_test PASSED in 16.3s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 9.9s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 21.5s //tensorflow/python/saved_model/registration:registration_test PASSED in 9.2s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 23.3s //tensorflow/python/saved_model/tests:variable_wrapper_test PASSED in 13.6s //tensorflow/python/summary:plugin_asset_test PASSED in 9.2s //tensorflow/python/summary:summary_iterator_test PASSED in 11.4s //tensorflow/python/summary:summary_test PASSED in 11.5s //tensorflow/python/summary:summary_v2_test PASSED in 11.6s //tensorflow/python/summary/writer:writer_test PASSED in 20.0s //tensorflow/python/tools:aot_compiled_test PASSED in 22.3s //tensorflow/python/tools:freeze_graph_test PASSED in 11.7s //tensorflow/python/tools:optimize_for_inference_test PASSED in 10.8s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 22.4s //tensorflow/python/tools:saved_model_cli_test PASSED in 34.8s //tensorflow/python/tools:saved_model_utils_test PASSED in 12.7s //tensorflow/python/tools:strip_unused_test PASSED in 13.2s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 13.7s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 18.4s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 30.6s //tensorflow/python/tools/api/generator2/extractor:extractor_test PASSED in 0.8s //tensorflow/python/tools/api/generator2/generator:generator_test PASSED in 0.7s //tensorflow/python/tools/api/generator2/shared:exported_api_test PASSED in 10.0s //tensorflow/python/tpu:bfloat16_test PASSED in 32.7s //tensorflow/python/tpu:feature_column_test PASSED in 36.4s //tensorflow/python/tpu:topology_test PASSED in 11.8s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 20.0s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 12.3s //tensorflow/python/tpu:tpu_embedding_v3_utils_test PASSED in 12.4s //tensorflow/python/tpu:tpu_infeed_test PASSED in 11.6s //tensorflow/python/tpu:tpu_sharding_test PASSED in 9.4s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 9.8s //tensorflow/python/tpu/client:client_py_test PASSED in 11.4s //tensorflow/python/trackable:autotrackable_test PASSED in 15.6s //tensorflow/python/trackable:base_delegate_test PASSED in 13.4s //tensorflow/python/trackable:base_test PASSED in 11.8s //tensorflow/python/trackable:python_state_test PASSED in 9.3s //tensorflow/python/trackable:resource_test PASSED in 10.5s //tensorflow/python/trackable:trackable_utils_test PASSED in 21.9s //tensorflow/python/training:adadelta_test_cpu PASSED in 22.2s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 11.9s //tensorflow/python/training:adagrad_test_cpu PASSED in 16.3s //tensorflow/python/training:adam_test_cpu PASSED in 20.3s //tensorflow/python/training:basic_loops_test_cpu PASSED in 12.6s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 26.0s //tensorflow/python/training:checkpoint_ops_test PASSED in 17.1s //tensorflow/python/training:coordinator_test_cpu PASSED in 18.4s //tensorflow/python/training:device_setter_test_cpu PASSED in 12.4s //tensorflow/python/training:ftrl_test_cpu PASSED in 22.5s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 15.8s //tensorflow/python/training:input_test PASSED in 32.5s //tensorflow/python/training:momentum_test_cpu PASSED in 18.8s //tensorflow/python/training:monitored_session_test PASSED in 42.7s //tensorflow/python/training:moving_averages_test_cpu PASSED in 18.5s //tensorflow/python/training:optimizer_test_cpu PASSED in 14.6s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 12.7s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 12.7s //tensorflow/python/training:quantize_training_test_cpu PASSED in 9.0s //tensorflow/python/training:queue_runner_test_cpu PASSED in 12.6s //tensorflow/python/training:rmsprop_test_cpu PASSED in 30.1s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 16.0s //tensorflow/python/training:saver_test_2gpu PASSED in 43.1s //tensorflow/python/training:saver_test_cpu PASSED in 58.6s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 10.2s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 10.1s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 11.1s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 12.1s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 9.7s //tensorflow/python/training:server_lib_test PASSED in 20.0s //tensorflow/python/training:session_manager_test_cpu PASSED in 80.5s //tensorflow/python/training:slot_creator_test_cpu PASSED in 17.3s //tensorflow/python/training:supervisor_test PASSED in 19.8s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 10.0s //tensorflow/python/training:training_ops_test_cpu PASSED in 10.7s //tensorflow/python/training:training_util_test PASSED in 13.1s //tensorflow/python/training:warm_starting_util_test PASSED in 31.6s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 36.1s //tensorflow/python/training/experimental:loss_scale_test PASSED in 42.0s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 18.8s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 12.2s //tensorflow/python/util:compat_test PASSED in 9.9s //tensorflow/python/util:decorator_utils_test PASSED in 10.2s //tensorflow/python/util:deprecation_test PASSED in 10.0s //tensorflow/python/util:dispatch_test PASSED in 12.8s //tensorflow/python/util:example_parser_configuration_test PASSED in 9.2s //tensorflow/python/util:fast_module_type_test PASSED in 9.7s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 9.3s //tensorflow/python/util:function_utils_test PASSED in 9.1s //tensorflow/python/util:keyword_args_test PASSED in 12.1s //tensorflow/python/util:lazy_loader_test PASSED in 9.2s //tensorflow/python/util:lock_util_test PASSED in 14.4s //tensorflow/python/util:module_wrapper_test PASSED in 10.6s //tensorflow/python/util:nest_test PASSED in 28.1s //tensorflow/python/util:object_identity_test PASSED in 10.1s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 3.1s //tensorflow/python/util:serialization_test PASSED in 9.2s //tensorflow/python/util:tf_contextlib_test PASSED in 19.7s //tensorflow/python/util:tf_decorator_test PASSED in 11.8s //tensorflow/python/util:tf_export_test PASSED in 11.2s //tensorflow/python/util:tf_inspect_test PASSED in 10.8s //tensorflow/python/util:tf_should_use_test PASSED in 10.0s //tensorflow/python/util:tf_stack_test PASSED in 19.7s //tensorflow/python/util:traceback_utils_test PASSED in 9.5s //tensorflow/python/util:type_annotations_test PASSED in 10.2s //tensorflow/python/util:variable_utils_test PASSED in 11.0s //tensorflow/python/util:vlog_test PASSED in 29.0s //tensorflow/python/util/protobuf:protobuf_compare_test PASSED in 4.4s //tensorflow/tools/api/tests:module_test PASSED in 28.3s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 18.3s //tensorflow/tools/common:public_api_test PASSED in 3.2s //tensorflow/tools/common:traverse_test PASSED in 3.0s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 26.6s //tensorflow/tools/compatibility:ast_edits_test PASSED in 10.5s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 23.6s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 25.1s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 9.4s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 9.8s //tensorflow/tools/docs:tf_doctest_test PASSED in 1.9s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.4s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 2.3s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 1.8s //tensorflow/tools/graph_transforms:transforms_test PASSED in 5.6s //tensorflow/tools/proto_splitter:merge_test PASSED in 0.2s //tensorflow/tools/proto_splitter:split_graph_def_test PASSED in 26.2s //tensorflow/tools/proto_splitter:split_test PASSED in 11.1s //tensorflow/tools/proto_splitter:util_test PASSED in 8.7s //tensorflow/tools/proto_splitter/cc:composable_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:graph_def_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:saved_model_splitter_test PASSED in 0.4s //tensorflow/tools/proto_splitter/cc:util_test PASSED in 5.3s //tensorflow/tools/proto_splitter/python:saved_model_test PASSED in 10.7s //tensorflow/tools/proto_splitter/python:test_util_test PASSED in 9.4s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.1s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 1.2s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 11.5s Stats over 2 runs: max = 11.5s, min = 10.8s, avg = 11.2s, dev = 0.3s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 11.5s Stats over 2 runs: max = 11.5s, min = 10.5s, avg = 11.0s, dev = 0.5s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 44.6s Stats over 2 runs: max = 44.6s, min = 36.6s, avg = 40.6s, dev = 4.0s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 46.7s Stats over 2 runs: max = 46.7s, min = 45.6s, avg = 46.1s, dev = 0.6s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 52.5s Stats over 2 runs: max = 52.5s, min = 46.9s, avg = 49.7s, dev = 2.8s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 43.4s Stats over 2 runs: max = 43.4s, min = 40.3s, avg = 41.9s, dev = 1.5s //tensorflow/python/data/kernel_tests:window_test PASSED in 49.3s Stats over 2 runs: max = 49.3s, min = 36.8s, avg = 43.1s, dev = 6.2s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 17.7s Stats over 2 runs: max = 17.7s, min = 17.6s, avg = 17.7s, dev = 0.0s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 20.3s Stats over 2 runs: max = 20.3s, min = 19.1s, avg = 19.7s, dev = 0.6s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 35.7s Stats over 2 runs: max = 35.7s, min = 33.3s, avg = 34.5s, dev = 1.2s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 35.7s Stats over 2 runs: max = 35.7s, min = 24.2s, avg = 29.9s, dev = 5.7s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 113.7s Stats over 2 runs: max = 113.7s, min = 40.6s, avg = 77.1s, dev = 36.6s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 9.9s Stats over 2 runs: max = 9.9s, min = 9.9s, avg = 9.9s, dev = 0.0s //tensorflow/python/ops:control_flow_ops_test_cpu PASSED in 49.3s Stats over 2 runs: max = 49.3s, min = 44.4s, avg = 46.8s, dev = 2.4s //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 29.1s Stats over 3 runs: max = 29.1s, min = 29.1s, avg = 29.1s, dev = 0.0s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 14.2s Stats over 3 runs: max = 14.2s, min = 13.4s, avg = 13.9s, dev = 0.4s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.2s Stats over 3 runs: max = 0.2s, min = 0.1s, avg = 0.1s, dev = 0.0s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 24.0s Stats over 3 runs: max = 24.0s, min = 16.2s, avg = 21.1s, dev = 3.5s //tensorflow/python/data/kernel_tests:unique_test PASSED in 17.2s Stats over 3 runs: max = 17.2s, min = 14.1s, avg = 15.6s, dev = 1.2s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 22.5s Stats over 3 runs: max = 22.5s, min = 18.1s, avg = 20.0s, dev = 1.8s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 55.2s Stats over 3 runs: max = 55.2s, min = 30.9s, avg = 39.6s, dev = 11.1s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 17.7s Stats over 3 runs: max = 17.7s, min = 17.0s, avg = 17.4s, dev = 0.3s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 39.1s Stats over 3 runs: max = 39.1s, min = 37.7s, avg = 38.2s, dev = 0.6s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 295.5s Stats over 3 runs: max = 295.5s, min = 15.9s, avg = 109.2s, dev = 131.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 27.5s Stats over 3 runs: max = 27.5s, min = 26.3s, avg = 26.8s, dev = 0.5s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 15.3s Stats over 3 runs: max = 15.3s, min = 12.2s, avg = 13.3s, dev = 1.4s //tensorflow/core/grappler/optimizers:mkl_remapper_test FAILED in 3 out of 3 in 13.9s Stats over 3 runs: max = 13.9s, min = 10.1s, avg = 12.2s, dev = 1.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/core/grappler/optimizers/mkl_remapper_test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/core/grappler/optimizers/mkl_remapper_test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/core/grappler/optimizers/mkl_remapper_test/test_attempts/attempt_2.log //tensorflow/core/kernels:example_parsing_ops_test PASSED in 0.5s Stats over 4 runs: max = 0.5s, min = 0.4s, avg = 0.5s, dev = 0.1s //tensorflow/dtensor/python/tests:batchparallel_spmd_test_cpu PASSED in 24.2s Stats over 4 runs: max = 24.2s, min = 23.3s, avg = 23.7s, dev = 0.3s //tensorflow/dtensor/python/tests:conv_test_cpu PASSED in 21.7s Stats over 4 runs: max = 21.7s, min = 19.7s, avg = 20.9s, dev = 0.7s //tensorflow/dtensor/python/tests:sparse_test_cpu PASSED in 13.7s Stats over 4 runs: max = 13.7s, min = 11.0s, avg = 12.1s, dev = 1.0s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 32.4s Stats over 4 runs: max = 32.4s, min = 17.8s, avg = 25.8s, dev = 5.5s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 49.5s Stats over 4 runs: max = 49.5s, min = 33.7s, avg = 38.1s, dev = 6.6s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 38.0s Stats over 4 runs: max = 38.0s, min = 13.4s, avg = 25.7s, dev = 11.8s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 34.3s Stats over 4 runs: max = 34.3s, min = 16.4s, avg = 24.3s, dev = 6.6s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 93.2s Stats over 4 runs: max = 93.2s, min = 86.0s, avg = 89.1s, dev = 3.1s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 11.7s Stats over 4 runs: max = 11.7s, min = 10.4s, avg = 11.0s, dev = 0.5s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 85.4s Stats over 4 runs: max = 85.4s, min = 61.1s, avg = 72.2s, dev = 10.0s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 29.6s Stats over 4 runs: max = 29.6s, min = 12.5s, avg = 20.5s, dev = 8.0s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 28.3s Stats over 4 runs: max = 28.3s, min = 17.0s, avg = 23.0s, dev = 4.4s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 19.5s Stats over 4 runs: max = 19.5s, min = 14.7s, avg = 16.8s, dev = 1.8s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 40.7s Stats over 4 runs: max = 40.7s, min = 37.6s, avg = 38.9s, dev = 1.3s //tensorflow/python/data/kernel_tests:skip_test PASSED in 57.9s Stats over 4 runs: max = 57.9s, min = 42.4s, avg = 49.8s, dev = 6.8s //tensorflow/python/data/kernel_tests:take_test PASSED in 58.1s Stats over 4 runs: max = 58.1s, min = 48.6s, avg = 53.1s, dev = 3.8s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 56.0s Stats over 4 runs: max = 56.0s, min = 41.0s, avg = 45.1s, dev = 6.3s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 22.7s Stats over 4 runs: max = 22.7s, min = 16.5s, avg = 19.6s, dev = 2.7s //tensorflow/python/data/kernel_tests:zip_test PASSED in 17.8s Stats over 4 runs: max = 17.8s, min = 16.4s, avg = 17.2s, dev = 0.5s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 19.4s Stats over 4 runs: max = 19.4s, min = 18.1s, avg = 18.6s, dev = 0.5s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 34.7s Stats over 4 runs: max = 34.7s, min = 25.4s, avg = 29.8s, dev = 3.7s //tensorflow/python/framework:convert_to_constants_test PASSED in 77.5s Stats over 4 runs: max = 77.5s, min = 64.4s, avg = 70.2s, dev = 5.1s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 41.3s Stats over 4 runs: max = 41.3s, min = 37.8s, avg = 39.8s, dev = 1.3s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 18.8s Stats over 4 runs: max = 18.8s, min = 12.6s, avg = 14.9s, dev = 2.3s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 84.3s Stats over 4 runs: max = 84.3s, min = 35.8s, avg = 60.1s, dev = 22.7s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 42.0s Stats over 4 runs: max = 42.0s, min = 17.5s, avg = 27.2s, dev = 10.1s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 113.8s Stats over 4 runs: max = 113.8s, min = 16.7s, avg = 58.3s, dev = 40.3s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 73.3s Stats over 4 runs: max = 73.3s, min = 72.5s, avg = 73.0s, dev = 0.3s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 49.4s Stats over 4 runs: max = 49.4s, min = 33.4s, avg = 40.9s, dev = 7.2s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 124.4s Stats over 4 runs: max = 124.4s, min = 11.0s, avg = 57.0s, dev = 47.3s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 25.9s Stats over 4 runs: max = 25.9s, min = 22.2s, avg = 24.0s, dev = 1.5s //tensorflow/python/ops:nn_batchnorm_test_cpu PASSED in 25.2s Stats over 4 runs: max = 25.2s, min = 18.9s, avg = 21.9s, dev = 2.8s //tensorflow/python/ops:nn_fused_batchnorm_d9m_test_cpu PASSED in 25.4s Stats over 4 runs: max = 25.4s, min = 20.8s, avg = 23.3s, dev = 1.7s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 110.6s Stats over 4 runs: max = 110.6s, min = 46.2s, avg = 75.4s, dev = 23.4s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 80.5s Stats over 4 runs: max = 80.5s, min = 73.6s, avg = 77.6s, dev = 2.6s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 16.4s Stats over 5 runs: max = 16.4s, min = 10.3s, avg = 13.0s, dev = 2.6s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 16.3s Stats over 5 runs: max = 16.3s, min = 10.4s, avg = 12.9s, dev = 2.2s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 21.2s Stats over 5 runs: max = 21.2s, min = 15.6s, avg = 18.0s, dev = 2.5s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 19.0s Stats over 5 runs: max = 19.0s, min = 15.3s, avg = 17.1s, dev = 1.5s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 10.7s Stats over 5 runs: max = 10.7s, min = 8.3s, avg = 9.6s, dev = 0.9s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 11.3s Stats over 5 runs: max = 11.3s, min = 10.6s, avg = 10.9s, dev = 0.2s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 18.5s Stats over 5 runs: max = 18.5s, min = 16.5s, avg = 17.3s, dev = 0.7s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 20.5s Stats over 5 runs: max = 20.5s, min = 18.9s, avg = 19.5s, dev = 0.6s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 141.4s Stats over 5 runs: max = 141.4s, min = 26.2s, avg = 67.9s, dev = 39.8s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 94.8s Stats over 5 runs: max = 94.8s, min = 17.8s, avg = 46.7s, dev = 26.2s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 4.0s Stats over 5 runs: max = 4.0s, min = 2.5s, avg = 3.1s, dev = 0.6s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 24.9s Stats over 5 runs: max = 24.9s, min = 23.6s, avg = 24.1s, dev = 0.5s //tensorflow/dtensor/python/tests:multi_mesh_test_cpu PASSED in 11.8s Stats over 5 runs: max = 11.8s, min = 10.6s, avg = 11.0s, dev = 0.4s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 15.9s Stats over 5 runs: max = 15.9s, min = 14.3s, avg = 15.2s, dev = 0.5s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 16.5s Stats over 5 runs: max = 16.5s, min = 14.6s, avg = 15.6s, dev = 0.6s //tensorflow/python/distribute:vars_test_2gpu PASSED in 22.6s Stats over 5 runs: max = 22.6s, min = 20.3s, avg = 21.9s, dev = 0.8s //tensorflow/python/distribute:vars_test_cpu PASSED in 48.2s Stats over 5 runs: max = 48.2s, min = 21.8s, avg = 40.6s, dev = 10.0s //tensorflow/python/eager:device_placement_test_cpu PASSED in 12.7s Stats over 5 runs: max = 12.7s, min = 10.8s, avg = 11.6s, dev = 0.7s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 183.8s Stats over 5 runs: max = 183.8s, min = 38.8s, avg = 88.5s, dev = 50.2s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 18.0s Stats over 5 runs: max = 18.0s, min = 12.1s, avg = 15.0s, dev = 2.4s //tensorflow/python/grappler:cluster_test_cpu PASSED in 11.7s Stats over 5 runs: max = 11.7s, min = 10.7s, avg = 11.1s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 63.2s Stats over 5 runs: max = 63.2s, min = 41.4s, avg = 53.0s, dev = 7.6s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 72.3s Stats over 5 runs: max = 72.3s, min = 61.2s, avg = 65.9s, dev = 3.6s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 120.0s Stats over 5 runs: max = 120.0s, min = 114.8s, avg = 118.2s, dev = 1.9s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 49.7s Stats over 5 runs: max = 49.7s, min = 45.5s, avg = 46.8s, dev = 1.6s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 89.0s Stats over 5 runs: max = 89.0s, min = 81.7s, avg = 86.2s, dev = 2.7s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 48.8s Stats over 5 runs: max = 48.8s, min = 47.3s, avg = 48.2s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 102.5s Stats over 5 runs: max = 102.5s, min = 96.2s, avg = 98.9s, dev = 2.1s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 39.8s Stats over 5 runs: max = 39.8s, min = 38.4s, avg = 39.2s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 38.6s Stats over 5 runs: max = 38.6s, min = 36.3s, avg = 37.3s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 52.5s Stats over 5 runs: max = 52.5s, min = 47.9s, avg = 49.2s, dev = 1.7s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 10.7s Stats over 5 runs: max = 10.7s, min = 10.4s, avg = 10.5s, dev = 0.1s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 55.1s Stats over 5 runs: max = 55.1s, min = 52.5s, avg = 53.7s, dev = 1.0s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 132.4s Stats over 5 runs: max = 132.4s, min = 8.7s, avg = 33.8s, dev = 49.3s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 16.7s Stats over 5 runs: max = 16.7s, min = 10.9s, avg = 12.5s, dev = 2.2s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 39.5s Stats over 5 runs: max = 39.5s, min = 31.2s, avg = 34.2s, dev = 2.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 38.0s Stats over 5 runs: max = 38.0s, min = 12.1s, avg = 19.2s, dev = 9.6s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 73.5s Stats over 5 runs: max = 73.5s, min = 45.6s, avg = 55.0s, dev = 9.6s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 33.9s Stats over 6 runs: max = 33.9s, min = 31.4s, avg = 32.4s, dev = 1.0s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 42.5s Stats over 6 runs: max = 42.5s, min = 19.4s, avg = 34.9s, dev = 7.2s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 24.6s Stats over 6 runs: max = 24.6s, min = 10.3s, avg = 16.4s, dev = 6.1s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 49.6s Stats over 6 runs: max = 49.6s, min = 12.6s, avg = 21.4s, dev = 12.8s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 53.7s Stats over 6 runs: max = 53.7s, min = 23.3s, avg = 35.8s, dev = 9.3s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 13.7s Stats over 7 runs: max = 13.7s, min = 9.6s, avg = 11.1s, dev = 1.4s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 11.5s Stats over 8 runs: max = 11.5s, min = 6.5s, avg = 8.1s, dev = 1.7s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 60.2s Stats over 8 runs: max = 60.2s, min = 10.4s, avg = 26.7s, dev = 17.0s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 67.8s Stats over 8 runs: max = 67.8s, min = 10.0s, avg = 30.2s, dev = 20.7s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 26.9s Stats over 8 runs: max = 26.9s, min = 13.0s, avg = 18.8s, dev = 4.8s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 28.8s Stats over 8 runs: max = 28.8s, min = 12.7s, avg = 21.9s, dev = 6.1s //tensorflow/dtensor/python/tests:input_util_test PASSED in 28.9s Stats over 8 runs: max = 28.9s, min = 20.3s, avg = 25.1s, dev = 2.8s //tensorflow/dtensor/python/tests:save_restore_v2_test_cpu PASSED in 22.5s Stats over 8 runs: max = 22.5s, min = 13.4s, avg = 16.4s, dev = 3.5s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 33.1s Stats over 8 runs: max = 33.1s, min = 10.2s, avg = 18.9s, dev = 8.1s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 53.1s Stats over 8 runs: max = 53.1s, min = 23.6s, avg = 37.9s, dev = 9.7s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 46.3s Stats over 8 runs: max = 46.3s, min = 13.1s, avg = 27.8s, dev = 13.6s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 36.4s Stats over 8 runs: max = 36.4s, min = 11.1s, avg = 18.9s, dev = 9.2s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 40.2s Stats over 8 runs: max = 40.2s, min = 8.9s, avg = 18.7s, dev = 11.8s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_load_ft_test PASSED in 45.7s Stats over 8 runs: max = 45.7s, min = 40.1s, avg = 42.5s, dev = 1.8s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_load_test PASSED in 87.4s Stats over 8 runs: max = 87.4s, min = 74.0s, avg = 80.3s, dev = 4.0s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 37.5s Stats over 8 runs: max = 37.5s, min = 5.7s, avg = 19.1s, dev = 12.5s //tensorflow/python/data/kernel_tests:batch_test PASSED in 26.4s Stats over 8 runs: max = 26.4s, min = 18.7s, avg = 23.0s, dev = 2.7s //tensorflow/python/data/kernel_tests:filter_test PASSED in 40.8s Stats over 8 runs: max = 40.8s, min = 21.6s, avg = 34.6s, dev = 6.2s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 52.3s Stats over 8 runs: max = 52.3s, min = 30.3s, avg = 40.4s, dev = 7.0s //tensorflow/python/data/kernel_tests:shard_test PASSED in 30.5s Stats over 8 runs: max = 30.5s, min = 20.1s, avg = 26.1s, dev = 3.3s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 57.3s Stats over 8 runs: max = 57.3s, min = 32.3s, avg = 36.6s, dev = 7.8s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 45.0s Stats over 8 runs: max = 45.0s, min = 16.7s, avg = 26.6s, dev = 9.8s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 88.3s Stats over 8 runs: max = 88.3s, min = 14.1s, avg = 36.2s, dev = 28.8s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 62.7s Stats over 8 runs: max = 62.7s, min = 43.1s, avg = 53.3s, dev = 6.8s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 189.4s Stats over 8 runs: max = 189.4s, min = 128.0s, avg = 156.6s, dev = 22.3s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 117.4s Stats over 8 runs: max = 117.4s, min = 99.8s, avg = 108.8s, dev = 6.2s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 56.7s Stats over 8 runs: max = 56.7s, min = 7.1s, avg = 16.4s, dev = 16.0s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 10.5s Stats over 8 runs: max = 10.5s, min = 7.8s, avg = 8.8s, dev = 0.8s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 54.9s Stats over 8 runs: max = 54.9s, min = 36.0s, avg = 43.5s, dev = 6.3s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 46.8s Stats over 8 runs: max = 46.8s, min = 31.9s, avg = 36.4s, dev = 4.2s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 11.3s Stats over 10 runs: max = 11.3s, min = 7.6s, avg = 9.7s, dev = 1.4s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 12.1s Stats over 10 runs: max = 12.1s, min = 7.5s, avg = 10.0s, dev = 1.4s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 15.9s Stats over 10 runs: max = 15.9s, min = 9.3s, avg = 12.3s, dev = 2.2s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 16.9s Stats over 10 runs: max = 16.9s, min = 10.7s, avg = 13.4s, dev = 1.9s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 82.4s Stats over 10 runs: max = 82.4s, min = 57.2s, avg = 67.1s, dev = 10.4s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 95.0s Stats over 10 runs: max = 95.0s, min = 43.9s, avg = 65.3s, dev = 15.9s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 22.3s Stats over 10 runs: max = 22.3s, min = 10.4s, avg = 14.3s, dev = 3.8s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 24.4s Stats over 10 runs: max = 24.4s, min = 11.5s, avg = 17.5s, dev = 4.9s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 22.0s Stats over 10 runs: max = 22.0s, min = 7.1s, avg = 14.5s, dev = 5.1s //tensorflow/python/framework:function_test_cpu PASSED in 53.7s Stats over 10 runs: max = 53.7s, min = 7.9s, avg = 14.2s, dev = 13.5s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 15.9s Stats over 10 runs: max = 15.9s, min = 10.2s, avg = 12.7s, dev = 1.8s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 24.7s Stats over 10 runs: max = 24.7s, min = 6.2s, avg = 17.1s, dev = 8.1s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 14.6s Stats over 10 runs: max = 14.6s, min = 9.8s, avg = 11.6s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 97.7s Stats over 10 runs: max = 97.7s, min = 90.1s, avg = 94.3s, dev = 2.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 67.1s Stats over 10 runs: max = 67.1s, min = 15.7s, avg = 42.9s, dev = 17.3s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 9.3s Stats over 10 runs: max = 9.3s, min = 8.0s, avg = 8.6s, dev = 0.4s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 32.7s Stats over 10 runs: max = 32.7s, min = 9.3s, avg = 22.8s, dev = 9.6s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 42.8s Stats over 10 runs: max = 42.8s, min = 8.8s, avg = 26.0s, dev = 16.1s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 26.9s Stats over 10 runs: max = 26.9s, min = 10.1s, avg = 14.2s, dev = 5.9s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 15.4s Stats over 10 runs: max = 15.4s, min = 12.6s, avg = 14.1s, dev = 0.8s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 13.6s Stats over 10 runs: max = 13.6s, min = 13.0s, avg = 13.2s, dev = 0.2s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 139.6s Stats over 10 runs: max = 139.6s, min = 20.3s, avg = 77.5s, dev = 55.6s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 50.3s Stats over 10 runs: max = 50.3s, min = 9.2s, avg = 16.0s, dev = 11.6s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 14.6s Stats over 10 runs: max = 14.6s, min = 9.0s, avg = 11.5s, dev = 1.9s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 121.5s Stats over 10 runs: max = 121.5s, min = 115.0s, avg = 119.2s, dev = 1.9s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 23.7s Stats over 10 runs: max = 23.7s, min = 20.0s, avg = 21.1s, dev = 1.1s //tensorflow/python/saved_model:load_test_cpu PASSED in 80.1s Stats over 10 runs: max = 80.1s, min = 31.2s, avg = 56.5s, dev = 14.0s //tensorflow/python/distribute/failure_handling:failure_handler_test FLAKY, failed in 2 out of 10 in 77.6s Stats over 10 runs: max = 77.6s, min = 35.9s, avg = 55.6s, dev = 10.9s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_8_of_8/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_5_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:fft_test_cpu PASSED in 22.9s Stats over 12 runs: max = 22.9s, min = 7.5s, avg = 14.8s, dev = 5.1s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 24.3s Stats over 12 runs: max = 24.3s, min = 11.7s, avg = 17.2s, dev = 4.1s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 23.8s Stats over 12 runs: max = 23.8s, min = 9.3s, avg = 15.0s, dev = 4.0s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 23.6s Stats over 12 runs: max = 23.6s, min = 6.4s, avg = 15.3s, dev = 7.1s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 43.5s Stats over 12 runs: max = 43.5s, min = 14.6s, avg = 20.3s, dev = 7.5s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 22.0s Stats over 12 runs: max = 22.0s, min = 14.0s, avg = 18.0s, dev = 2.4s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 249.0s Stats over 12 runs: max = 249.0s, min = 35.1s, avg = 72.7s, dev = 58.9s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 243.9s Stats over 12 runs: max = 243.9s, min = 30.6s, avg = 70.5s, dev = 59.6s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 22.5s Stats over 15 runs: max = 22.5s, min = 14.7s, avg = 17.7s, dev = 2.4s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 131.7s Stats over 15 runs: max = 131.7s, min = 112.4s, avg = 120.8s, dev = 6.3s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 57.4s Stats over 15 runs: max = 57.4s, min = 13.3s, avg = 21.5s, dev = 11.0s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 17.5s Stats over 16 runs: max = 17.5s, min = 8.9s, avg = 12.8s, dev = 2.7s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 32.5s Stats over 16 runs: max = 32.5s, min = 12.5s, avg = 21.4s, dev = 5.5s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 35.7s Stats over 16 runs: max = 35.7s, min = 12.1s, avg = 16.2s, dev = 5.6s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 11.3s Stats over 16 runs: max = 11.3s, min = 5.5s, avg = 7.3s, dev = 1.7s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 14.3s Stats over 16 runs: max = 14.3s, min = 8.5s, avg = 11.4s, dev = 1.8s //tensorflow/python/ops:image_ops_test_cpu PASSED in 51.8s Stats over 16 runs: max = 51.8s, min = 42.0s, avg = 46.2s, dev = 2.8s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 107.5s Stats over 17 runs: max = 107.5s, min = 7.0s, avg = 44.6s, dev = 29.3s //tensorflow/python/data/kernel_tests:map_test PASSED in 73.2s Stats over 19 runs: max = 73.2s, min = 14.1s, avg = 49.7s, dev = 14.8s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 11.6s Stats over 20 runs: max = 11.6s, min = 6.1s, avg = 8.9s, dev = 1.4s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 10.9s Stats over 20 runs: max = 10.9s, min = 6.2s, avg = 8.2s, dev = 1.4s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 13.8s Stats over 20 runs: max = 13.8s, min = 8.8s, avg = 11.2s, dev = 1.1s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 21.3s Stats over 20 runs: max = 21.3s, min = 8.7s, avg = 11.6s, dev = 2.7s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 11.3s Stats over 20 runs: max = 11.3s, min = 6.0s, avg = 8.9s, dev = 1.6s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 25.4s Stats over 20 runs: max = 25.4s, min = 8.7s, avg = 13.0s, dev = 4.5s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 28.5s Stats over 20 runs: max = 28.5s, min = 3.9s, avg = 9.8s, dev = 6.6s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 20.1s Stats over 20 runs: max = 20.1s, min = 12.5s, avg = 16.1s, dev = 2.5s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 36.5s Stats over 20 runs: max = 36.5s, min = 33.1s, avg = 34.8s, dev = 0.7s //tensorflow/python/kernel_tests:metrics_test PASSED in 56.0s Stats over 20 runs: max = 56.0s, min = 7.4s, avg = 22.8s, dev = 14.9s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 10.7s Stats over 20 runs: max = 10.7s, min = 4.2s, avg = 7.4s, dev = 2.4s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 31.8s Stats over 20 runs: max = 31.8s, min = 3.5s, avg = 9.6s, dev = 9.2s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 56.6s Stats over 20 runs: max = 56.6s, min = 4.4s, avg = 23.9s, dev = 15.4s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 107.3s Stats over 20 runs: max = 107.3s, min = 26.3s, avg = 59.9s, dev = 23.7s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 11.0s Stats over 20 runs: max = 11.0s, min = 9.1s, avg = 10.0s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 31.9s Stats over 20 runs: max = 31.9s, min = 23.0s, avg = 27.8s, dev = 3.0s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 139.8s Stats over 20 runs: max = 139.8s, min = 39.7s, avg = 90.8s, dev = 33.5s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 28.8s Stats over 20 runs: max = 28.8s, min = 5.9s, avg = 14.0s, dev = 6.8s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 28.2s Stats over 20 runs: max = 28.2s, min = 14.3s, avg = 20.3s, dev = 4.9s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 22.5s Stats over 20 runs: max = 22.5s, min = 17.3s, avg = 19.2s, dev = 1.3s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 75.8s Stats over 20 runs: max = 75.8s, min = 8.5s, avg = 34.7s, dev = 24.3s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 24.9s Stats over 20 runs: max = 24.9s, min = 9.8s, avg = 14.2s, dev = 3.3s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 27.8s Stats over 24 runs: max = 27.8s, min = 8.5s, avg = 16.8s, dev = 5.4s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 40.9s Stats over 24 runs: max = 40.9s, min = 13.5s, avg = 25.3s, dev = 6.1s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 39.6s Stats over 25 runs: max = 39.6s, min = 28.4s, avg = 35.2s, dev = 3.3s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 25.3s Stats over 30 runs: max = 25.3s, min = 4.0s, avg = 10.1s, dev = 5.2s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 22.8s Stats over 32 runs: max = 22.8s, min = 5.0s, avg = 11.0s, dev = 4.6s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 29.3s Stats over 32 runs: max = 29.3s, min = 10.4s, avg = 20.6s, dev = 5.9s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 3.5s Stats over 48 runs: max = 3.5s, min = 0.7s, avg = 1.6s, dev = 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 83.2s Stats over 50 runs: max = 83.2s, min = 37.6s, avg = 54.6s, dev = 10.0s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 22.2s Stats over 50 runs: max = 22.2s, min = 4.0s, avg = 12.1s, dev = 3.9s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 25.5s Stats over 50 runs: max = 25.5s, min = 4.1s, avg = 13.1s, dev = 6.1s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 66.3s Stats over 50 runs: max = 66.3s, min = 34.1s, avg = 44.2s, dev = 7.5s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 14.6s Stats over 50 runs: max = 14.6s, min = 4.6s, avg = 8.2s, dev = 2.2s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 32.5s Stats over 50 runs: max = 32.5s, min = 8.2s, avg = 15.8s, dev = 6.3s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 17.0s Stats over 50 runs: max = 17.0s, min = 4.1s, avg = 6.1s, dev = 2.1s Executed 3055 out of 3055 tests: 3054 tests pass and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.