==================== Test output for //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test: 2023-11-24 21:46:01.795495: W external/local_tsl/tsl/lib/monitoring/collection_registry.cc:81] Trying to register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay. The old value will be erased in order to register a new one. Please check if you link the metric more than once, or if the name is already used by other metrics. Running main() from gmock_main.cc [==========] Running 13 tests from 1 test suite. [----------] Global test environment set-up. [----------] 13 tests from ConvertTfQuantToMhloIntTest [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizeAndDequantize WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1700862361.912502 254365 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862362.056483 254365 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformQuantizeAndDequantize (241 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizePerChannel I0000 00:00:1700862362.149551 254365 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862362.181753 254365 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformQuantizePerChannel (50 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformDequantizePerChannel I0000 00:00:1700862362.199146 254365 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862362.223660 254365 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformDequantizePerChannel (41 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizeConvolution I0000 00:00:1700862362.241101 254365 cpu_client.cc:370] TfrtCpuClient created. loc("-":13:8): error: Out-of-bounds dimension 281474237449952, expected to be less than the input-tensor rank 4 F0000 00:00:1700862362.253819 254365 convert_tf_quant_to_mhlo_int_test.cc:193] Check failed: succeeded(pm.run(module_op.get())) *** Check failure stack trace: *** @ 0xffff9b3acea0 absl::lts_20230802::log_internal::LogMessage::SendToLog() @ 0xffff9b3ad60c absl::lts_20230802::log_internal::LogMessageFatal::~LogMessageFatal() @ 0xaaaab302d640 mlir::quant::stablehlo::(anonymous namespace)::ConvertTfQuantToMhloIntTest::ExecuteAndCompareResultsWithTfKernel() @ 0xaaaab302e7e8 mlir::quant::stablehlo::(anonymous namespace)::ConvertTfQuantToMhloIntTest_UniformQuantizeConvolution_Test::TestBody() @ 0xffff82082f0c testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0xffff82083228 testing::Test::Run() @ 0xffff82083604 testing::TestInfo::Run() @ 0xffff82083fb8 testing::TestSuite::Run() @ 0xffff8208b37c testing::internal::UnitTestImpl::RunAllTests() @ 0xffff82083990 testing::UnitTest::Run() @ 0xffff95070880 main @ 0xffff7ff00e10 __libc_start_main @ 0xaaaab3022f1c (unknown) ================================================================================ ==================== Test output for //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test: 2023-11-24 21:47:25.813449: W external/local_tsl/tsl/lib/monitoring/collection_registry.cc:81] Trying to register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay. The old value will be erased in order to register a new one. Please check if you link the metric more than once, or if the name is already used by other metrics. Running main() from gmock_main.cc [==========] Running 13 tests from 1 test suite. [----------] Global test environment set-up. [----------] 13 tests from ConvertTfQuantToMhloIntTest [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizeAndDequantize WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1700862445.856322 340477 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862445.921311 340477 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformQuantizeAndDequantize (106 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizePerChannel I0000 00:00:1700862445.956097 340477 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862445.992778 340477 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformQuantizePerChannel (57 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformDequantizePerChannel I0000 00:00:1700862446.012789 340477 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862446.041630 340477 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformDequantizePerChannel (219 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizeConvolution I0000 00:00:1700862446.231513 340477 cpu_client.cc:370] TfrtCpuClient created. loc("-":13:8): error: Out-of-bounds dimension 281474803423936, expected to be less than the input-tensor rank 4 F0000 00:00:1700862446.245285 340477 convert_tf_quant_to_mhlo_int_test.cc:193] Check failed: succeeded(pm.run(module_op.get())) *** Check failure stack trace: *** @ 0xffff7ddecea0 absl::lts_20230802::log_internal::LogMessage::SendToLog() @ 0xffff7dded60c absl::lts_20230802::log_internal::LogMessageFatal::~LogMessageFatal() @ 0xaaaad678d640 mlir::quant::stablehlo::(anonymous namespace)::ConvertTfQuantToMhloIntTest::ExecuteAndCompareResultsWithTfKernel() @ 0xaaaad678e7e8 mlir::quant::stablehlo::(anonymous namespace)::ConvertTfQuantToMhloIntTest_UniformQuantizeConvolution_Test::TestBody() @ 0xffff64ac2f0c testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0xffff64ac3228 testing::Test::Run() @ 0xffff64ac3604 testing::TestInfo::Run() @ 0xffff64ac3fb8 testing::TestSuite::Run() @ 0xffff64acb37c testing::internal::UnitTestImpl::RunAllTests() @ 0xffff64ac3990 testing::UnitTest::Run() @ 0xffff77ab0880 main @ 0xffff62940e10 __libc_start_main @ 0xaaaad6782f1c (unknown) ================================================================================ ==================== Test output for //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test: 2023-11-24 21:47:48.344775: W external/local_tsl/tsl/lib/monitoring/collection_registry.cc:81] Trying to register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay. The old value will be erased in order to register a new one. Please check if you link the metric more than once, or if the name is already used by other metrics. Running main() from gmock_main.cc [==========] Running 13 tests from 1 test suite. [----------] Global test environment set-up. [----------] 13 tests from ConvertTfQuantToMhloIntTest [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizeAndDequantize WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1700862468.387466 354035 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862468.461766 354035 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformQuantizeAndDequantize (99 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizePerChannel I0000 00:00:1700862468.480610 354035 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862468.515392 354035 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformQuantizePerChannel (54 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformDequantizePerChannel I0000 00:00:1700862468.535357 354035 cpu_client.cc:370] TfrtCpuClient created. I0000 00:00:1700862468.564832 354035 cpu_client.cc:373] TfrtCpuClient destroyed. [ OK ] ConvertTfQuantToMhloIntTest.UniformDequantizePerChannel (56 ms) [ RUN ] ConvertTfQuantToMhloIntTest.UniformQuantizeConvolution I0000 00:00:1700862468.590827 354035 cpu_client.cc:370] TfrtCpuClient created. loc("-":13:8): error: Out-of-bounds dimension 281474858418352, expected to be less than the input-tensor rank 4 F0000 00:00:1700862468.604326 354035 convert_tf_quant_to_mhlo_int_test.cc:193] Check failed: succeeded(pm.run(module_op.get())) *** Check failure stack trace: *** @ 0xffffa023cea0 absl::lts_20230802::log_internal::LogMessage::SendToLog() @ 0xffffa023d60c absl::lts_20230802::log_internal::LogMessageFatal::~LogMessageFatal() @ 0xaaaad322d640 mlir::quant::stablehlo::(anonymous namespace)::ConvertTfQuantToMhloIntTest::ExecuteAndCompareResultsWithTfKernel() @ 0xaaaad322e7e8 mlir::quant::stablehlo::(anonymous namespace)::ConvertTfQuantToMhloIntTest_UniformQuantizeConvolution_Test::TestBody() @ 0xffff86f12f0c testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0xffff86f13228 testing::Test::Run() @ 0xffff86f13604 testing::TestInfo::Run() @ 0xffff86f13fb8 testing::TestSuite::Run() @ 0xffff86f1b37c testing::internal::UnitTestImpl::RunAllTests() @ 0xffff86f13990 testing::UnitTest::Run() @ 0xffff99f00880 main @ 0xffff84d90e10 __libc_start_main @ 0xaaaad3222f1c (unknown) ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 5 of 8): Running tests under Python 3.10.13: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I1124 21:56:22.371846 281473168276128 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I1124 21:56:22.372344 281473168276128 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W1124 21:56:22.372689 281473168276128 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I1124 21:56:22.372915 281473168276128 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee615edd0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1124 21:56:22.574657 281473168276128 polymorphic_function.py:157] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee615edd0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee615edd0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1124 21:56:22.590408 281473168276128 polymorphic_function.py:157] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee615edd0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I1124 21:56:22.727200 281473168276128 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I1124 21:56:22.960686 281473168276128 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I1124 21:56:23.277083 281473168276128 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I1124 21:56:23.532355 281473168276128 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I1124 21:56:23.764782 281473168276128 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I1124 21:56:23.998832 281473168276128 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I1124 21:56:24.237798 281473168276128 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:sending sigterm I1124 21:56:24.316205 281470244876768 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I1124 21:56:24.335073 281473168276128 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:Termination caught in main thread on preempted worker I1124 21:56:24.335941 281473168276128 failure_handling.py:1159] Termination caught in main thread on preempted worker INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. I1124 21:56:24.358821 281473168276128 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpmddw6f7c/fh_ckpt I1124 21:56:24.408898 281473168276128 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpmddw6f7c/fh_ckpt INFO:tensorflow:Continue training for the grace period. I1124 21:56:24.409295 281473168276128 failure_handling.py:1134] Continue training for the grace period. INFO:tensorflow:epoch 7 finished I1124 21:56:24.565079 281473168276128 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I1124 21:56:24.565797 281473168276128 failure_handler_test.py:245] Training finished. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 2.28s I1124 21:56:24.566932 281473168276128 test_util.py:2544] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice): 2.28s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_checkpoint_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24078 I1124 21:56:24.578635 281473168276128 test_util.py:3887] Using local port 24078 INFO:tensorflow:Using local port 24077 I1124 21:56:24.580809 281473168276128 test_util.py:3887] Using local port 24077 INFO:tensorflow:Using local port 24076 I1124 21:56:24.582760 281473168276128 test_util.py:3887] Using local port 24076 INFO:tensorflow:Using local port 24075 I1124 21:56:24.584640 281473168276128 test_util.py:3887] Using local port 24075 INFO:tensorflow:Cluster starting. I1124 21:56:28.930193 281473168276128 failure_handler_test.py:297] Cluster starting. [worker-0]: I1124 21:56:29.061905 281473182890656 multi_process_runner.py:840] Subprocess with PID 1304273 (worker, 0) is now being started. [worker-0]: I1124 21:56:29.062411 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1124 21:56:29.208141 281473182890656 multi_process_runner.py:840] Subprocess with PID 1304294 (worker, 1) is now being started. [worker-1]: I1124 21:56:29.208618 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I1124 21:56:29.212477 281473182890656 multi_process_runner.py:840] Subprocess with PID 1304365 (worker, 2) is now being started. [worker-2]: I1124 21:56:29.212974 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I1124 21:56:29.221769 281473182890656 multi_process_runner.py:840] Subprocess with PID 1304370 (worker, 3) is now being started. [worker-3]: I1124 21:56:29.222245 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-11-24 21:56:29.289541: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24078 [worker-0]: 2023-11-24 21:56:29.327849: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 1331021824073900460 [worker-0]: 2023-11-24 21:56:29.329015: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2023-11-24 21:56:29.420133: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24077 [worker-0]: 2023-11-24 21:56:29.422614: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 9534300706356017111 [worker-1]: 2023-11-24 21:56:29.423034: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-11-24 21:56:29.506672: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24076 [worker-3]: 2023-11-24 21:56:29.516999: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24075 [worker-0]: 2023-11-24 21:56:29.519639: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 8483119502978815354 [worker-3]: 2023-11-24 21:56:29.519841: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:29.526490: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 14610454676983027016 [worker-2]: 2023-11-24 21:56:29.528041: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I1124 21:56:29.530020 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1124 21:56:29.530486 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I1124 21:56:29.530026 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1124 21:56:29.544965 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I1124 21:56:29.584259 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1124 21:56:29.584832 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1124 21:56:29.585070 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1124 21:56:29.584260 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1124 21:56:29.584831 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1124 21:56:29.585068 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I1124 21:56:29.592037 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I1124 21:56:29.592714 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:29.592957 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1124 21:56:29.641141 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1124 21:56:29.641756 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1124 21:56:29.641993 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1124 21:56:29.746974 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:29.750818 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I1124 21:56:29.756288 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I1124 21:56:29.765366 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1124 21:56:29.776743 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1124 21:56:29.777146 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:29.787539 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1124 21:56:29.787900 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: I1124 21:56:29.787747 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1124 21:56:29.788095 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1124 21:56:29.788456 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W1124 21:56:29.777499 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: W1124 21:56:29.788253 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-1]: Instructions for updating: [worker-3]: INFO:tensorflow:Start training at 0 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: I1124 21:56:29.788664 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: I1124 21:56:29.777711 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-1]: I1124 21:56:29.788479 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1124 21:56:29.836526 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I1124 21:56:29.836934 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W1124 21:56:29.837309 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I1124 21:56:29.837524 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:30.050933 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:30.065557 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:30.099056 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:30.154349 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:30.300243 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:30.307636 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:30.323011 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:30.316530 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:30.427441 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:30.448365 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:30.450390 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:30.456550 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:30.527542 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:30.542645 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:30.539744 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:30.556659 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:30.658146 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:30.677752 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:30.673073 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:30.682600 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:30.798403 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901ec550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:30.807371 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901ec550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:30.813006 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:30.818432 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:30.829860 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:30.836517 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:30.843066 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:30.892981 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:31.039244 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:31.046502 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:31.057796 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901ecc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:31.068798 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901ecc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:31.067387 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:31.075518 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:31.110617 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:31.132520 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:31.234898 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:31.252884 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:31.257662 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:31.278093 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:31.432592 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:31.432710 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:31.432652 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:31.482003 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:31.683749 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:31.696738 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:31.714743 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:31.699980 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:31.872742 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:31.888448 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:31.889044 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:31.903098 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.012448 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:32.062803 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.062693 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.088210 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.185221 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.193986 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.185644 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:32.192872 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.312900 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.301923 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:32.316593 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.312931 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:32.428358 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.429212 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.442997 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.473505 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: I1124 21:56:32.558137 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-1]: I1124 21:56:32.558465 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I1124 21:56:32.559154 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I1124 21:56:32.561717 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.571415 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.574350 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.593011 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:32.602273 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.703977 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.727496 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.752488 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:32.738246 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.851243 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.864049 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:32.878131 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.882724 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:32.982707 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:32.995504 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:32.995909 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:33.018388 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I1124 21:56:33.066662 281473168276128 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I1124 21:56:33.067154 281473168276128 failure_handler_test.py:306] sigterm sent [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:33.118886 281473182890656 failure_handling.py:710] Member 2 has received termination notice. [worker-3]: I1124 21:56:33.119369 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I1124 21:56:33.119682 281473182890656 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:33.120404 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 20 [worker-2]: I1124 21:56:33.137971 281473182890656 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 20 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I1124 21:56:33.142374 281446849638880 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-0]: I1124 21:56:33.144403 281448090890720 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: I1124 21:56:33.144327 281473182890656 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I1124 21:56:33.145428 281473182890656 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-1]: I1124 21:56:33.141357 281446858093024 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: I1124 21:56:33.148040 281473182890656 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I1124 21:56:33.149747 281473182890656 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:33.153055 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:33.148885 281448619307488 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:33.161477 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I1124 21:56:33.227326 281473182890656 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I1124 21:56:33.248903 281473182890656 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I1124 21:56:33.251082 281473182890656 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I1124 21:56:33.247815 281473182890656 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/fh_ckpt [worker-0]: I1124 21:56:33.298946 281473182890656 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/fh_ckpt [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I1124 21:56:33.304933 281473182890656 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I1124 21:56:33.305305 281473182890656 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/workertemp_1/fh_ckpt [worker-1]: I1124 21:56:33.316867 281473182890656 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/workertemp_1/fh_ckpt [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/workertemp_2/fh_ckpt [worker-2]: I1124 21:56:33.321343 281473182890656 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/workertemp_2/fh_ckpt [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/workertemp_3/fh_ckpt [worker-3]: I1124 21:56:33.323531 281473182890656 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/e874e4537dc68e6784aaca6594115ad2axhyayok/tmpq36ssugo/workertemp_3/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I1124 21:56:33.330070 281473182890656 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: I1124 21:56:33.330013 281473182890656 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I1124 21:56:33.330500 281473182890656 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I1124 21:56:33.333380 281473182890656 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I1124 21:56:33.333714 281473182890656 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I1124 21:56:33.330383 281473182890656 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I1124 21:56:35.080293 281473168276128 failure_handler_test.py:309] restarting workers INFO:tensorflow:workers restarted I1124 21:56:35.276579 281473168276128 failure_handler_test.py:313] workers restarted [worker-0]: I1124 21:56:35.398539 281473182890656 multi_process_runner.py:840] Subprocess with PID 1314713 (worker, 0) is now being started. [worker-0]: I1124 21:56:35.399007 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1124 21:56:35.507929 281473182890656 multi_process_runner.py:840] Subprocess with PID 1314777 (worker, 1) is now being started. [worker-1]: I1124 21:56:35.508386 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I1124 21:56:35.677628 281473182890656 multi_process_runner.py:840] Subprocess with PID 1314974 (worker, 3) is now being started. [worker-2]: I1124 21:56:35.675161 281473182890656 multi_process_runner.py:840] Subprocess with PID 1314949 (worker, 2) is now being started. [worker-3]: I1124 21:56:35.678087 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I1124 21:56:35.675691 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24078", "localhost:24077", "localhost:24076", "localhost:24075"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-2]: 2023-11-24 21:56:35.862429: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24076 [worker-3]: 2023-11-24 21:56:35.940918: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24075 [worker-0]: 2023-11-24 21:56:35.947618: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24078 [worker-0]: 2023-11-24 21:56:35.961270: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 819210751164640726 [worker-2]: 2023-11-24 21:56:35.962355: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:35.961463: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 17003136280652875062 [worker-3]: 2023-11-24 21:56:35.966857: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:35.994979: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 6560673683242912740 [worker-0]: 2023-11-24 21:56:35.995428: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2023-11-24 21:56:36.195898: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24077 [worker-0]: 2023-11-24 21:56:36.306526: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2767770139018588710 [worker-1]: 2023-11-24 21:56:36.326501: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I1124 21:56:36.331788 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1124 21:56:36.332665 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I1124 21:56:36.331516 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1124 21:56:36.347800 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1124 21:56:36.387214 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1124 21:56:36.387745 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I1124 21:56:36.389146 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I1124 21:56:36.389649 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:36.389879 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: I1124 21:56:36.387975 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1124 21:56:36.396974 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1124 21:56:36.397661 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1124 21:56:36.397906 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1124 21:56:36.414864 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1124 21:56:36.415410 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1124 21:56:36.415643 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24078', 'localhost:24077', 'localhost:24076', 'localhost:24075']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:36.603236 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1124 21:56:36.605363 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I1124 21:56:36.605736 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W1124 21:56:36.606152 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 20 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:36.606369 281473182890656 failure_handler_test.py:197] Start training at 20 [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1124 21:56:36.618363 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I1124 21:56:36.614341 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:36.615202 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1124 21:56:36.615463 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: INFO:tensorflow:training restarted [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1124 21:56:36.622488 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-3]: I1124 21:56:36.626638 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: Instructions for updating: [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1124 21:56:36.620698 281473182890656 failure_handler_test.py:207] training restarted [worker-3]: I1124 21:56:36.627511 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1124 21:56:36.627791 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: I1124 21:56:36.622869 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: W1124 21:56:36.615838 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1124 21:56:36.628191 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 20 [worker-3]: I1124 21:56:36.628408 281473182890656 failure_handler_test.py:197] Start training at 20 [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 20 [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W1124 21:56:36.623281 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 20 [worker-0]: I1124 21:56:36.623502 281473182890656 failure_handler_test.py:197] Start training at 20 [worker-1]: I1124 21:56:36.616052 281473182890656 failure_handler_test.py:197] Start training at 20 [worker-0]: INFO:tensorflow:training restarted [worker-0]: I1124 21:56:36.639881 281473182890656 failure_handler_test.py:207] training restarted [worker-1]: INFO:tensorflow:training restarted [worker-1]: I1124 21:56:36.618301 281473182890656 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:training restarted [worker-3]: I1124 21:56:36.663787 281473182890656 failure_handler_test.py:207] training restarted [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:36.834882 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:36.881521 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:36.857199 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:36.906316 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:36.978859 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:36.978868 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:36.998394 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.003845 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.129555 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.130058 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.171134 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.182351 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.274851 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.288290 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.287446 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.285795 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.360802 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.360726 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.379328 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.397992 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901f05e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901f05e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901f45e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901f45e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:37.449694 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901f05e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:37.453505 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901f45e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:37.449747 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901f45e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.462149 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W1124 21:56:37.449481 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901f05e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.462303 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.468360 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.483034 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901f4ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901f4ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:37.548073 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901f4ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901f0ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:37.547070 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901f4ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:37.547315 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901f0ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901f0ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:37.547707 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901f0ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.559871 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.560111 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.560352 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.560543 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.629909 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.634753 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.646579 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.653159 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.718537 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.719895 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.726810 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.719715 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.786380 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.787321 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.787320 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.787459 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I1124 21:56:37.839699 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-0]: I1124 21:56:37.839860 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-1]: I1124 21:56:37.839943 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-2]: I1124 21:56:37.840012 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.852163 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.852264 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.852515 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.853413 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.914256 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.914287 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.914366 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.914706 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.973774 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.974323 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.975248 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.975275 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.063212 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.093937 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.142484 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.347733 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.458405 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.453484 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.467180 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.491641 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.590162 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.603983 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.618890 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.612035 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.751695 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.750145 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.773087 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.805484 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.947669 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.948645 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.959237 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.977894 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:39.089145 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:39.079732 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:39.120091 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:39.300717 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:39.431453 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:39.441924 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:39.444703 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:39.502721 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:39.616235 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:39.620066 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:39.642984 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:39.644269 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:39.793518 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:39.797249 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:39.817029 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:39.843821 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:39.959049 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:39.958425 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:39.967113 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:39.967538 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:40.053133 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:40.047033 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:40.071722 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:40.084003 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:40.350608 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:40.350496 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:40.371766 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:40.367287 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I1124 21:56:40.447041 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I1124 21:56:40.452004 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I1124 21:56:40.455445 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-2]: I1124 21:56:40.454840 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:40.466424 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:40.470622 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:40.493153 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:40.522974 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:40.791894 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:40.804054 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:40.803366 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:40.803960 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:40.869516 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:40.882536 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:40.902163 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:40.912106 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.010974 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.012973 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.025382 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.016632 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.087401 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.093173 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.107412 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.097109 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.172683 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.175619 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.176536 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.182044 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.246454 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.281900 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.282218 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.277063 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.367824 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.385708 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.385410 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.391406 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.531774 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.561905 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.561767 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.592036 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.661116 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.663536 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.682193 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.683223 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.815079 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.832041 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.842738 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.840813 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.941750 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.934001 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.952109 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.019895 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.132880 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.137271 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.151746 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.147439 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.234699 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.242132 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.243066 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.261699 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.334240 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.361844 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.383980 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.438089 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I1124 21:56:42.626878 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I1124 21:56:42.641414 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I1124 21:56:42.663418 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.651976 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I1124 21:56:42.676638 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.671752 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.690904 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.688274 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.769243 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.781491 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.781771 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.807846 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.944743 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.947849 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.971778 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.004666 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.153437 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.162042 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.171843 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.182790 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.345798 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.341464 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.359523 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.373725 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.495527 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.502114 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.541916 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.577338 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.699222 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.717380 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.721711 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.742545 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.848359 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.847760 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.848701 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.868890 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.983888 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.994836 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.999847 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.989831 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.067546 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.072771 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.098651 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.099845 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.188995 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.191841 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.191775 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.266476 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.357681 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.365840 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.390342 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.392745 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.471798 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.474916 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.478779 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.483125 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.568251 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.578965 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.593141 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.593133 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.797256 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.821925 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.822710 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.811695 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I1124 21:56:44.927001 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I1124 21:56:44.940277 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I1124 21:56:44.956568 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I1124 21:56:44.957068 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.958568 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.953188 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.982286 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.990092 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.094598 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.113668 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.137259 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.162974 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.286738 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.301818 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.302153 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.331747 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.429017 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.443092 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.461299 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.477767 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.591484 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.591955 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.601933 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.603461 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.665473 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.691396 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.699989 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.816193 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.876288 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.880742 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.875730 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.884185 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.009517 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.055885 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.042242 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.078916 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.171791 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.177206 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.191418 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.189286 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.431141 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.425162 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.431699 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.451827 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.549331 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.554582 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.573490 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.577651 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.778129 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.781874 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.781831 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.791248 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.931787 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.948788 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.941746 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.932821 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.112034 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.111441 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.153526 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.167309 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.311567 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.353219 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.341799 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.343103 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-3]: I1124 21:56:47.469311 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:epoch 5 finished [worker-1]: I1124 21:56:47.479218 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-0]: I1124 21:56:47.477142 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.490801 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.494482 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-2]: I1124 21:56:47.486399 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.506616 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.497799 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.605066 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.604493 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.617974 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.637273 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.737812 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.761835 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.791903 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.807341 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.991760 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.007897 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.012616 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.016952 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.101219 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.096959 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.104094 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.096972 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.194790 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.211721 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.222130 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.225975 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.316334 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.323740 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.327353 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.349265 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.430551 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.437429 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.441805 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.444988 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.515668 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.519714 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.523070 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.523144 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.646549 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.648027 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.651495 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.678263 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.741521 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.741523 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.752694 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.753285 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.840304 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.861968 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.861809 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.904107 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.967345 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.967357 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.969908 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.970189 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.136813 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.139162 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.131557 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.162917 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.227879 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.228970 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.232187 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.238858 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-3]: I1124 21:56:49.294779 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: I1124 21:56:49.294944 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-2]: I1124 21:56:49.295197 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I1124 21:56:49.298339 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.306070 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.306833 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.306956 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.314116 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.387448 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.387516 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.387468 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.403080 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.479426 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.480557 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.485296 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.530549 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.644477 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.662052 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.678645 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.676584 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.787532 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.792743 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.796277 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.797979 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.881934 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.887425 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.887296 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.902582 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.984822 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.984880 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.989325 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.984840 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.058416 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.059017 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.063991 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.072627 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.135635 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.138669 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.152475 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.156837 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.244601 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.244017 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.259482 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.272982 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.348885 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.367072 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.386435 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.392003 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.470789 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.467221 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.486826 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.497551 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.621842 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.638260 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.639088 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.727671 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.911755 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.921803 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.927027 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.942321 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:51.068851 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:51.081655 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:51.102337 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:51.108126 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I1124 21:56:51.265012 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:epoch 7 finished [worker-0]: I1124 21:56:51.267484 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-1]: I1124 21:56:51.267662 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I1124 21:56:51.270303 281473182890656 failure_handler_test.py:245] Training finished. [worker-2]: I1124 21:56:51.269970 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I1124 21:56:51.271487 281473182890656 failure_handler_test.py:245] Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I1124 21:56:51.273705 281473182890656 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I1124 21:56:51.278187 281473182890656 failure_handler_test.py:245] Training finished. I1124 21:56:52.212021 281473168276128 multi_process_runner.py:646] worker-0 exit code: 0 I1124 21:56:52.212395 281473168276128 multi_process_runner.py:646] worker-1 exit code: 0 I1124 21:56:52.212573 281473168276128 multi_process_runner.py:646] worker-2 exit code: 0 I1124 21:56:52.212739 281473168276128 multi_process_runner.py:646] worker-3 exit code: 0 I1124 21:56:52.218490 281473168276128 multi_process_runner.py:662] Joining log reading threads. I1124 21:56:52.218968 281473168276128 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 27.94s I1124 21:56:52.508999 281473168276128 test_util.py:2544] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 27.94s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24066 I1124 21:56:52.513250 281473168276128 test_util.py:3887] Using local port 24066 INFO:tensorflow:Using local port 24065 I1124 21:56:52.515427 281473168276128 test_util.py:3887] Using local port 24065 INFO:tensorflow:Using local port 24064 I1124 21:56:52.517934 281473168276128 test_util.py:3887] Using local port 24064 INFO:tensorflow:Using local port 24063 I1124 21:56:52.520194 281473168276128 test_util.py:3887] Using local port 24063 INFO:tensorflow:Cluster starting. I1124 21:56:52.544270 281473168276128 failure_handler_test.py:297] Cluster starting. [worker-1]: I1124 21:56:52.852240 281473182890656 multi_process_runner.py:840] Subprocess with PID 1352884 (worker, 1) is now being started. [worker-1]: I1124 21:56:52.852685 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24066", "localhost:24065", "localhost:24064", "localhost:24063"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I1124 21:56:52.956521 281473182890656 multi_process_runner.py:840] Subprocess with PID 1352833 (worker, 0) is now being started. [worker-0]: I1124 21:56:52.957010 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24066", "localhost:24065", "localhost:24064", "localhost:24063"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I1124 21:56:53.052692 281473182890656 multi_process_runner.py:840] Subprocess with PID 1353842 (worker, 2) is now being started. [worker-2]: I1124 21:56:53.053133 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24066", "localhost:24065", "localhost:24064", "localhost:24063"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I1124 21:56:53.063728 281473182890656 multi_process_runner.py:840] Subprocess with PID 1354271 (worker, 3) is now being started. [worker-3]: I1124 21:56:53.064227 281473182890656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24066", "localhost:24065", "localhost:24064", "localhost:24063"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: 2023-11-24 21:56:53.121503: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24065 [worker-2]: 2023-11-24 21:56:53.366430: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24064 [worker-3]: 2023-11-24 21:56:53.473652: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24063 [worker-0]: 2023-11-24 21:56:53.513482: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24066 [worker-0]: 2023-11-24 21:56:53.543640: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 9767574464062929660 [worker-0]: 2023-11-24 21:56:53.546409: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:53.557222: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 13734248132032353787 [worker-3]: 2023-11-24 21:56:53.560360: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:53.567166: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 1909185713023186363 [worker-2]: 2023-11-24 21:56:53.567558: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:54.166215: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 4147149193431775331 [worker-1]: 2023-11-24 21:56:54.167471: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I1124 21:56:54.170915 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1124 21:56:54.177610 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I1124 21:56:54.171608 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1124 21:56:54.188620 281473182890656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I1124 21:56:54.232421 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I1124 21:56:54.233022 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:54.233262 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1124 21:56:54.243316 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1124 21:56:54.248287 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1124 21:56:54.248542 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I1124 21:56:54.332043 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1124 21:56:54.332707 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1124 21:56:54.332957 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1124 21:56:54.338461 281473182890656 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1124 21:56:54.339164 281473182890656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1124 21:56:54.339417 281473182890656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24066', 'localhost:24065', 'localhost:24064', 'localhost:24063']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:54.465374 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1124 21:56:54.467469 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I1124 21:56:54.467767 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W1124 21:56:54.468098 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I1124 21:56:54.468305 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I1124 21:56:54.500621 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I1124 21:56:54.488674 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:54.478384 281473182890656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1124 21:56:54.489408 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-3]: I1124 21:56:54.501388 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1124 21:56:54.489686 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: I1124 21:56:54.501654 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-3]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W1124 21:56:54.490037 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: W1124 21:56:54.502048 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-3]: I1124 21:56:54.502259 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-0]: I1124 21:56:54.490262 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:54.512178 281473182890656 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1124 21:56:54.512658 281473182890656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1124 21:56:54.513081 281473182890656 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I1124 21:56:54.513300 281473182890656 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:54.689465 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:54.735206 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:54.769334 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:54.770723 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:54.859820 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:54.877723 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:54.882500 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:54.901420 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:54.969973 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:54.970221 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.009734 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.033477 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.150265 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.151232 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.152546 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.179127 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.242351 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.243543 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.243198 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.242811 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901e8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901ec550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:55.293146 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901e4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff901e8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:55.293314 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901e8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:55.293565 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901ec550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:55.293476 281473182890656 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff901e8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.304822 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.305702 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.307307 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.310019 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901e8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:55.370943 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901e4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:55.371258 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901e8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901ecc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff901e8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:sending sigterm [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1124 21:56:56.896599 281473168276128 failure_handler_test.py:302] sending sigterm [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 19.6s [worker-0]: I1124 21:56:55.382924 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1124 21:57:12.105062 281473168276128 test_util.py:2544] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 19.6s [worker-2]: W1124 21:56:55.376592 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901ecc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I1124 21:56:55.382381 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.388863 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.450522 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.451532 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.513447 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.451737 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.514438 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.514188 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.577326 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker [worker-1]: W1124 21:56:55.371516 281473182890656 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff901e8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ====================================================================== ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='checkpoint', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- [worker-3]: I1124 21:56:55.577341 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process [worker-2]: I1124 21:56:55.577940 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ---------------------------------------------------------------------- [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 Ran 3 tests in 49.833s [worker-0]: I1124 21:56:55.637586 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 FAILED (errors=1) [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.383431 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.638844 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.638853 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.698722 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.759003 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.699673 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.820133 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.451642 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.700825 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.760247 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.881502 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.514892 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.821385 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.882750 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I1124 21:56:55.931837 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.944880 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.003047 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.062365 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.121351 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.178073 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.240212 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.304262 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.366570 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.428016 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.490061 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.551956 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.613849 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.676542 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.739029 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.796950 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I1124 21:56:56.843653 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.855443 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.912289 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.970214 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.029685 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.760922 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: I1124 21:56:55.577952 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.088649 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.639344 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.146717 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.700083 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.205890 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.760807 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.264371 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.822152 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.381811 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.932053 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-1]: I1124 21:56:55.883070 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.569854 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 0 finished [worker-0]: I1124 21:56:55.943130 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.932298 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.821930 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.627496 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.003171 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.944841 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.684864 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.062120 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.882435 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 0 finished [worker-1]: I1124 21:56:56.003075 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.932305 281473182890656 failure_handler_test.py:195] epoch 0 finished [worker-0]: I1124 21:56:56.120720 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.944216 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.178318 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.238470 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.061996 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.004046 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.739862 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.302150 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.064008 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.121769 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.798929 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.365701 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.120772 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.426246 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.179326 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.860186 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.179652 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: I1124 21:56:56.239702 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.488092 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.240378 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.908700 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-1]: I1124 21:56:56.304757 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.550395 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.304774 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.367298 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.612066 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.427913 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.921229 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.366663 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.674380 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.979845 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.428955 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.038852 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.736838 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.489749 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.097784 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.795592 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.551789 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.489868 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.152432 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.843818 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.614274 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.551846 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.209280 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.854404 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.614296 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.676435 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.265720 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.911312 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.676452 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.738727 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.321687 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.969005 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.796331 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.378700 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.435513 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.739452 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-0]: I1124 21:56:57.028240 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.843986 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.492701 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.854901 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.796368 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.087360 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I1124 21:56:58.549247 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.844037 281473182890656 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.145097 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.912671 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.605864 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.855002 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.204210 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.970540 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.911525 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.660826 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.262533 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.029447 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.969542 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.715928 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.367386 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-1]: I1124 21:56:57.089148 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.030332 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.568934 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.761481 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.147163 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.088215 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.626922 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.206349 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.772847 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.146701 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.264760 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.829631 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.205691 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.265093 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.397438 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.568775 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.626769 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.684289 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.740744 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.801640 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.887444 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.684355 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.861491 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I1124 21:56:57.909005 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-3]: I1124 21:56:58.944478 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.922760 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.002267 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.978958 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.059975 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.038781 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.113266 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.097224 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.168729 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.152990 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.222009 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.209184 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.276378 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.740976 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.372149 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.265192 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.334282 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.801602 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.568684 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.321739 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.862266 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.626795 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.378296 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.908853 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-1]: I1124 21:56:57.684313 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.436367 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.741947 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.391922 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.921301 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.494216 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.801553 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.449194 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.549999 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.979672 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.504326 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.862747 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-3]: I1124 21:56:59.558495 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.908999 281473182890656 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.603294 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-1]: I1124 21:56:57.922773 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.613680 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.980389 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.668079 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.037318 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.722603 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.097258 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.777589 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.153039 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.832097 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.209302 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.265368 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.321785 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.888080 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.378364 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.944326 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.436407 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.998285 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.492063 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.052536 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.549378 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.106583 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.606348 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.160865 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.661023 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.606931 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.661428 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.716682 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-0]: I1124 21:56:58.038770 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.761804 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.216877 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.716219 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: I1124 21:56:58.097328 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.772482 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.761609 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.153075 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.270758 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.828626 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.209278 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.771827 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.266678 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.886044 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.323043 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.943394 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.828642 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.378585 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.001249 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.886320 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.435224 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.059486 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.943403 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.492712 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.115625 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.000882 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.549471 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.059507 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.169380 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.606354 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.115533 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.222372 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.324693 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.661144 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.169313 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.276800 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.379154 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.716268 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-1]: I1124 21:56:59.222310 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.332632 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.423764 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.276687 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.761492 281473182890656 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.389819 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.433843 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.332652 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.771893 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.447676 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.389814 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.487482 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.828748 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.503643 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.541048 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.447623 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.886209 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.557835 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.596645 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-1]: I1124 21:56:59.503498 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.943536 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.603581 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.650345 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.557803 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.001386 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.613208 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.704027 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.603518 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-2]: I1124 21:56:59.667367 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.059583 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.613212 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.757657 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.722392 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.667359 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.776920 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.722025 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.812100 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.831423 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.776979 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.867909 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.887003 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.114281 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.831354 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.922250 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.943678 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.887039 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.169001 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.976676 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.998107 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.943636 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.222269 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.031165 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.052098 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.998133 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.276623 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.085232 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.105986 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.052125 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.332693 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.139393 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.160371 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.106143 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.389778 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.194587 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.215839 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.160407 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.447679 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.239544 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-1]: I1124 21:57:00.216305 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.270379 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.503531 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.252353 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.324343 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.270377 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.306539 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.557943 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.378761 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.324290 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.358781 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.603396 281473182890656 failure_handler_test.py:195] epoch 4 finished [worker-2]: I1124 21:57:00.424064 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-1]: I1124 21:57:00.378701 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-3]: I1124 21:57:01.413548 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.424023 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.613022 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.433620 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.667438 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.434130 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.487168 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.471730 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.722514 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.487136 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.540819 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.530494 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.776939 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.540817 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.596350 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.596358 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.831480 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.587468 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.650212 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.887063 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.650205 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.703700 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.644925 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.943660 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.703714 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.757407 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.701409 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.998342 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.757419 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.811569 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.052080 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.811481 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.867416 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.758288 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.106030 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.921598 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.815570 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.160479 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.873346 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.215932 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.932384 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.270476 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.324296 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:02.137313 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.378798 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:02.197612 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-3]: INFO:tensorflow:epoch 7 finished [worker-0]: I1124 21:57:00.423907 281473182890656 failure_handler_test.py:195] epoch 5 finished [worker-3]: I1124 21:57:02.244828 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Training finished. [worker-0]: I1124 21:57:00.434037 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:02.246046 281473182890656 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.487183 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.540894 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.596336 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.650315 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.703732 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.757443 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.867430 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.811592 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.921602 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.976668 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.867579 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.030146 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.921622 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.084619 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.976799 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.138824 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.030196 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.194281 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-0]: I1124 21:57:01.084750 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.239833 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.138900 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.251117 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.194496 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.306381 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.239615 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-2]: I1124 21:57:01.359476 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.250814 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.411875 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.306399 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.976695 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.470136 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.030706 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.528942 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.084636 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.585869 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.138873 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.642842 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.194315 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.699809 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.239768 281473182890656 failure_handler_test.py:195] epoch 6 finished [worker-2]: I1124 21:57:01.756623 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.250972 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.814045 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.306374 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.872082 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.359032 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.930913 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.411940 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.109431 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.470160 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.197856 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.528956 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.245098 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-1]: I1124 21:57:01.585868 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.246438 281473182890656 failure_handler_test.py:245] Training finished. [worker-1]: I1124 21:57:01.642814 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.699844 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.756631 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.814069 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.871716 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.930935 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:02.110578 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:02.197908 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I1124 21:57:02.245064 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I1124 21:57:02.246522 281473182890656 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.359486 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.411903 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.470214 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.529065 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.585943 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.642917 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.699877 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.756632 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.814093 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.872128 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.930983 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.142966 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.197493 281473182890656 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I1124 21:57:02.245008 281473182890656 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I1124 21:57:02.246690 281473182890656 failure_handler_test.py:245] Training finished. ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 1 of 8): Running tests under Python 3.10.13: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_error_propagation INFO:tensorflow:Using local port 24089 I1124 21:56:22.293773 281473177189024 test_util.py:3887] Using local port 24089 INFO:tensorflow:Using local port 24087 I1124 21:56:22.298053 281473177189024 test_util.py:3887] Using local port 24087 INFO:tensorflow:Using local port 24085 I1124 21:56:22.301456 281473177189024 test_util.py:3887] Using local port 24085 INFO:tensorflow:Using local port 24083 I1124 21:56:22.304725 281473177189024 test_util.py:3887] Using local port 24083 INFO:tensorflow:Cluster starting. I1124 21:56:26.902756 281473177189024 failure_handler_test.py:387] Cluster starting. [worker-0]: I1124 21:56:26.963726 281473073576608 multi_process_runner.py:840] Subprocess with PID 1297845 (worker, 0) is now being started. [worker-1]: I1124 21:56:26.968033 281473073576608 multi_process_runner.py:840] Subprocess with PID 1297912 (worker, 1) is now being started. [worker-0]: I1124 21:56:26.964143 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24089", "localhost:24087", "localhost:24085", "localhost:24083"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1124 21:56:26.968469 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24089", "localhost:24087", "localhost:24085", "localhost:24083"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-1]: 2023-11-24 21:56:27.197277: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24087 [worker-3]: I1124 21:56:27.223606 281473073576608 multi_process_runner.py:840] Subprocess with PID 1298083 (worker, 3) is now being started. [worker-2]: I1124 21:56:27.223604 281473073576608 multi_process_runner.py:840] Subprocess with PID 1298007 (worker, 2) is now being started. [worker-3]: I1124 21:56:27.224020 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24089", "localhost:24087", "localhost:24085", "localhost:24083"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I1124 21:56:27.224021 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24089", "localhost:24087", "localhost:24085", "localhost:24083"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: 2023-11-24 21:56:27.377369: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24089 [worker-0]: 2023-11-24 21:56:27.386905: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 9123993530730316434 [worker-0]: 2023-11-24 21:56:27.387585: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-11-24 21:56:27.443971: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24083 [worker-2]: 2023-11-24 21:56:27.446878: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24085 [worker-0]: 2023-11-24 21:56:27.449076: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 12385791245542670953 [worker-2]: 2023-11-24 21:56:27.449260: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:27.489551: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 9649975859585976545 [worker-3]: 2023-11-24 21:56:27.489774: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:28.206755: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2593712818799292714 [worker-1]: 2023-11-24 21:56:28.206977: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1124 21:56:28.208884 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1124 21:56:28.209162 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I1124 21:56:28.216290 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I1124 21:56:28.217053 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1124 21:56:28.266192 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1124 21:56:28.266720 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1124 21:56:28.266945 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1124 21:56:28.267739 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1124 21:56:28.268227 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1124 21:56:28.268450 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I1124 21:56:28.274351 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1124 21:56:28.274832 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1124 21:56:28.275055 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I1124 21:56:28.274126 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I1124 21:56:28.274636 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:28.274861 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24089', 'localhost:24087', 'localhost:24085', 'localhost:24083']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:28.336944 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1124 21:56:28.335831 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I1124 21:56:28.339416 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1124 21:56:28.336742 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-2]: I1124 21:56:28.339689 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I1124 21:56:28.336995 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: Instructions for updating: [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-2]: W1124 21:56:28.340019 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-0]: W1124 21:56:28.337319 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-2]: INFO:tensorflow:Start training at 0 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I1124 21:56:28.340226 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I1124 21:56:28.337524 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I1124 21:56:28.355803 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:28.358887 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1124 21:56:28.359262 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1124 21:56:28.359660 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I1124 21:56:28.359876 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I1124 21:56:28.355428 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I1124 21:56:28.356290 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1124 21:56:28.356712 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1124 21:56:28.357114 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I1124 21:56:28.357329 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:28.549893 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:28.580584 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Error reported to Coordinator: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: Traceback (most recent call last): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-2]: yield [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: I1124 21:56:28.611322 281447478456800 coordinator.py:213] Error reported to Coordinator: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-0]: 2023-11-24 21:56:28.618727: E external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:992] /job:worker/replica:0/task:2 has been set to ERROR in coordination service: RESOURCE_EXHAUSTED: in user code: [worker-2]: Traceback (most recent call last): [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-2]: yield [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: I1124 21:56:28.616791 281473073576608 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: 2023-11-24 21:56:28.626212: E external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:767] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-2]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: ResourceExhaustedError: Running out of resources [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-0]: 2023-11-24 21:56:28.626298: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-0]: 2023-11-24 21:56:28.626333: E tensorflow/core/common_runtime/ring_alg.cc:291] Aborting RingReduce with RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-1]: 2023-11-24 21:56:28.632597: E external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:767] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-0]: ResourceExhaustedError: Running out of resources [worker-3]: 2023-11-24 21:56:28.636495: E external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:767] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-3]: [worker-0]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: 2023-11-24 21:56:28.632678: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: 2023-11-24 21:56:28.642097: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-1]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: [worker-1]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-1]: File "", line 1, in [worker-0]: The error could be from a previous operation. Restart your program to reset. [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-2]: 2023-11-24 21:56:28.617137: E external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:767] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: in user code: [worker-0]: 2023-11-24 21:56:28.636253: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: [worker-3]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: ResourceExhaustedError: Running out of resources [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-2]: [worker-1]: [worker-3]: 2023-11-24 21:56:28.636545: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-2]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-0]: [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: ResourceExhaustedError: Running out of resources [worker-2]: 2023-11-24 21:56:28.617199: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: in user code: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: [worker-1]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-1]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: ResourceExhaustedError: Running out of resources [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-2]: [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: [worker-1]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: I1124 21:56:28.646458 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: [worker-3]: 2023-11-24 21:56:28.706331: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: RESOURCE_EXHAUSTED: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-2]: ResourceExhaustedError: Running out of resources [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-3]: [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: 2023-11-24 21:56:28.617233: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:438] Reporting error to coordination service: RESOURCE_EXHAUSTED: in user code: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-0]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-2]: [worker-0]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: ResourceExhaustedError: Running out of resources [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: [worker-0]: File "", line 1, in [worker-2]: [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-2]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: [worker-1]: [Op:__inference_train_step_38] [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-1]: I1124 21:56:28.672169 281473073576608 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-1]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: [type.googleapis.com/tensorflow.DerivedStatus=''] [worker-1]: [worker-3]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: File "", line 1, in [worker-3]: [worker-1]: [worker-3]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "", line 1, in [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: [worker-3]: [worker-1]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-1]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: ResourceExhaustedError: Running out of resources [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-0]: [worker-3]: [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-1]: [Op:__inference_train_step_38] [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-1]: I1124 21:56:28.672752 281473073576608 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-0]: [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: [Op:__inference_train_step_38] [worker-0]: [worker-3]: I1124 21:56:28.712267 281473073576608 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: [worker-0]: [worker-3]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-3]: [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: File "", line 1, in [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-3]: [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: [worker-0]: [Op:__inference_train_step_40] [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: I1124 21:56:28.643129 281473073576608 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-0]: [worker-3]: [worker-0]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: [worker-0]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: [worker-0]: [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-0]: [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: [Op:__inference_train_step_38] [worker-0]: [worker-3]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: I1124 21:56:28.712907 281473073576608 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-0]: [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-0]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: [Op:__inference_train_step_40] [worker-0]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-0]: I1124 21:56:28.643572 281473073576608 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. I1124 21:56:28.946318 281473177189024 multi_process_runner.py:646] worker-0 exit code: 0 I1124 21:56:28.946635 281473177189024 multi_process_runner.py:646] worker-1 exit code: 0 I1124 21:56:28.946811 281473177189024 multi_process_runner.py:646] worker-2 exit code: 0 I1124 21:56:28.946976 281473177189024 multi_process_runner.py:646] worker-3 exit code: 0 I1124 21:56:28.951125 281473177189024 multi_process_runner.py:662] Joining log reading threads. I1124 21:56:28.951399 281473177189024 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_error_propagation): 6.81s I1124 21:56:29.091501 281473177189024 test_util.py:2544] time(__main__.PreemptionCheckpointTest.test_error_propagation): 6.81s [ OK ] PreemptionCheckpointTest.test_error_propagation [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I1124 21:56:29.267518 281473177189024 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I1124 21:56:29.268025 281473177189024 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W1124 21:56:29.268407 281473177189024 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I1124 21:56:29.268632 281473177189024 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee68f2ef0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1124 21:56:29.480352 281473177189024 polymorphic_function.py:157] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xfffee68f2ef0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee68f2ef0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W1124 21:56:29.496787 281473177189024 polymorphic_function.py:157] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xfffee68f2ef0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I1124 21:56:29.676571 281473177189024 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I1124 21:56:30.168077 281473177189024 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I1124 21:56:30.415332 281473177189024 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I1124 21:56:30.622330 281473177189024 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I1124 21:56:30.866491 281473177189024 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I1124 21:56:31.132840 281473177189024 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:epoch 6 finished I1124 21:56:31.365333 281473177189024 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I1124 21:56:31.775539 281473177189024 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I1124 21:56:31.776280 281473177189024 failure_handler_test.py:245] Training finished. INFO:tensorflow:sending sigterm I1124 21:56:32.116234 281470244418016 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I1124 21:56:32.146270 281473177189024 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 3.05s I1124 21:56:32.147135 281473177189024 test_util.py:2544] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 3.05s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24074 I1124 21:56:32.151877 281473177189024 test_util.py:3887] Using local port 24074 INFO:tensorflow:Using local port 24073 I1124 21:56:32.153735 281473177189024 test_util.py:3887] Using local port 24073 INFO:tensorflow:Using local port 24072 I1124 21:56:32.155430 281473177189024 test_util.py:3887] Using local port 24072 INFO:tensorflow:Using local port 24071 I1124 21:56:32.157098 281473177189024 test_util.py:3887] Using local port 24071 INFO:tensorflow:Cluster starting. I1124 21:56:32.408815 281473177189024 failure_handler_test.py:297] Cluster starting. [worker-0]: I1124 21:56:32.668015 281473073576608 multi_process_runner.py:840] Subprocess with PID 1310550 (worker, 0) is now being started. [worker-1]: I1124 21:56:32.697861 281473073576608 multi_process_runner.py:840] Subprocess with PID 1310564 (worker, 1) is now being started. [worker-0]: I1124 21:56:32.668511 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1124 21:56:32.698331 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I1124 21:56:32.734273 281473073576608 multi_process_runner.py:840] Subprocess with PID 1310575 (worker, 2) is now being started. [worker-2]: I1124 21:56:32.734739 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I1124 21:56:32.798823 281473073576608 multi_process_runner.py:840] Subprocess with PID 1310586 (worker, 3) is now being started. [worker-3]: I1124 21:56:32.799309 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-11-24 21:56:32.968911: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24074 [worker-2]: 2023-11-24 21:56:32.991071: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24072 [worker-0]: 2023-11-24 21:56:33.010449: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 13850315885160177953 [worker-0]: 2023-11-24 21:56:33.010620: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 11372295865122036411 [worker-0]: 2023-11-24 21:56:33.010804: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-11-24 21:56:33.019931: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-11-24 21:56:33.116325: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24071 [worker-0]: 2023-11-24 21:56:33.126998: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 17540847533822770799 [worker-3]: 2023-11-24 21:56:33.143811: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2023-11-24 21:56:33.146400: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24073 [worker-0]: 2023-11-24 21:56:33.157522: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 15826279566945973515 [worker-1]: 2023-11-24 21:56:33.157728: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I1124 21:56:33.160824 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1124 21:56:33.160345 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1124 21:56:33.176919 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I1124 21:56:33.160592 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1124 21:56:33.263607 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1124 21:56:33.264133 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1124 21:56:33.264362 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I1124 21:56:33.275660 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-3]: I1124 21:56:33.280680 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I1124 21:56:33.276957 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Check health not enabled. [worker-0]: I1124 21:56:33.277201 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:33.281199 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:33.281425 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1124 21:56:33.285223 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1124 21:56:33.285749 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1124 21:56:33.285975 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:33.378542 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I1124 21:56:33.387002 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I1124 21:56:33.400187 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1124 21:56:33.403950 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I1124 21:56:33.416364 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1124 21:56:33.416693 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1124 21:56:33.417031 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I1124 21:56:33.417235 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-2]: I1124 21:56:33.405413 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I1124 21:56:33.405760 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1124 21:56:33.458065 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1124 21:56:33.458456 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W1124 21:56:33.458797 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I1124 21:56:33.459004 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W1124 21:56:33.406090 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I1124 21:56:33.406313 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:33.486496 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1124 21:56:33.486867 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1124 21:56:33.487218 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I1124 21:56:33.487424 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:33.719382 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:33.720447 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:33.727775 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:33.831860 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:33.900318 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:33.902115 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:33.917166 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:33.918744 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.051115 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.061635 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:34.062155 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.051139 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:34.166691 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.188328 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.189807 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.221646 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.321206 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.329223 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.333153 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:34.351888 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:34.414933 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W1124 21:56:34.426855 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I1124 21:56:34.427689 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840fc550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:34.436528 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840fc550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840f4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:34.437230 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840f4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.438345 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.464675 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.481734 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840f4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:34.566826 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840f4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:34.566709 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.578072 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:34.585336 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840fcc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:34.586408 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840fcc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:34.591004 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.596615 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.628119 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:34.710528 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.721296 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.721379 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.749511 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.857601 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.861882 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:34.865930 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.872257 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:34.965101 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:34.968190 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:34.973299 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:34.978054 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:35.186455 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:35.203126 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:35.232714 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:35.237341 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:35.314309 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:35.317515 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:35.324758 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:35.341703 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:35.436285 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:35.433216 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:35.461786 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:35.481151 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:35.753018 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:35.762570 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:35.751727 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:35.751715 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:35.913187 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:35.932241 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:35.953662 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:35.983519 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I1124 21:56:36.092069 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I1124 21:56:36.093606 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I1124 21:56:36.094392 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I1124 21:56:36.096479 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:36.105341 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:36.112666 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:36.116746 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:36.131700 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:36.220784 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:36.234959 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:36.236954 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:36.242460 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:36.371854 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:36.383056 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:36.401820 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:36.402794 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:36.622085 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:36.622989 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:36.631854 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:36.651793 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.033004 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.042045 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.041996 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.047527 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.161735 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.167667 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.169168 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.173109 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.231323 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.231388 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.231390 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.256846 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.332959 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.336736 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.337631 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.336379 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.408740 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.409355 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.425434 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.426532 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.512712 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.508030 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.521711 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.527004 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.590848 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.598536 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.597059 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.598425 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.667574 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.670900 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.672241 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.681884 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.768154 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.777306 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.780307 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.781496 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.845291 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.845308 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.846658 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.848125 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.906938 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.908200 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.908379 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.908798 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-3]: I1124 21:56:37.959166 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-2]: I1124 21:56:37.959397 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-0]: I1124 21:56:37.959425 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-1]: I1124 21:56:37.959369 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:37.971906 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:37.972395 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:37.972917 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:37.973046 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.035735 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.037034 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.048443 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.051453 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.120711 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.137164 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.123011 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.146673 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.212756 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.213639 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.212800 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.217213 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.291715 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.292108 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.287221 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.292138 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.408912 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.408787 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.411627 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.432848 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.551565 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.561105 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.565043 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.582080 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.649058 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.658022 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I1124 21:56:38.668301 281473177189024 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I1124 21:56:38.668785 281473177189024 failure_handler_test.py:306] sigterm sent [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-2]: I1124 21:56:38.676476 281473073576608 failure_handling.py:710] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.685526 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.673623 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:38.757909 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I1124 21:56:38.767678 281473073576608 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-0]: I1124 21:56:38.769918 281447075803616 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: I1124 21:56:38.769926 281449382867424 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 39 [worker-2]: I1124 21:56:38.770505 281473073576608 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 39 [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-2]: I1124 21:56:38.771491 281473073576608 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: I1124 21:56:38.788147 281473073576608 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-1]: I1124 21:56:38.788674 281450817516000 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I1124 21:56:38.790326 281473073576608 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:38.788177 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:38.792802 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-2]: I1124 21:56:38.791715 281473073576608 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:38.802411 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-3]: I1124 21:56:38.791009 281448191816160 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I1124 21:56:38.867267 281473073576608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I1124 21:56:38.867514 281473073576608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I1124 21:56:38.866891 281473073576608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I1124 21:56:38.867076 281473073576608 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/workertemp_3/fh_ckpt [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/workertemp_1/fh_ckpt [worker-3]: I1124 21:56:38.912174 281473073576608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/workertemp_3/fh_ckpt [worker-1]: I1124 21:56:38.911798 281473073576608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/workertemp_1/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I1124 21:56:38.913350 281473073576608 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I1124 21:56:38.913573 281473073576608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/fh_ckpt [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/workertemp_2/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I1124 21:56:38.933542 281473073576608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/fh_ckpt [worker-2]: I1124 21:56:38.944180 281473073576608 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b90092y_cai/tmptb_j7674/workertemp_2/fh_ckpt [worker-3]: I1124 21:56:38.943753 281473073576608 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I1124 21:56:38.944065 281473073576608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: I1124 21:56:38.949720 281473073576608 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I1124 21:56:38.950008 281473073576608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I1124 21:56:38.977404 281473073576608 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I1124 21:56:38.977763 281473073576608 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I1124 21:56:40.676260 281473177189024 failure_handler_test.py:309] restarting workers INFO:tensorflow:workers restarted I1124 21:56:40.770770 281473177189024 failure_handler_test.py:313] workers restarted [worker-0]: I1124 21:56:40.808130 281473073576608 multi_process_runner.py:840] Subprocess with PID 1327162 (worker, 0) is now being started. [worker-0]: I1124 21:56:40.808582 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1124 21:56:40.881928 281473073576608 multi_process_runner.py:840] Subprocess with PID 1327171 (worker, 1) is now being started. [worker-1]: I1124 21:56:40.882408 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I1124 21:56:40.897078 281473073576608 multi_process_runner.py:840] Subprocess with PID 1327238 (worker, 2) is now being started. [worker-3]: I1124 21:56:40.897552 281473073576608 multi_process_runner.py:840] Subprocess with PID 1327338 (worker, 3) is now being started. [worker-2]: I1124 21:56:40.897530 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I1124 21:56:40.897942 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24074", "localhost:24073", "localhost:24072", "localhost:24071"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2023-11-24 21:56:41.036688: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24072 [worker-0]: 2023-11-24 21:56:41.056938: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24074 [worker-0]: 2023-11-24 21:56:41.120633: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 985172139272974077 [worker-2]: 2023-11-24 21:56:41.121389: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2023-11-24 21:56:41.128402: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24073 [worker-0]: 2023-11-24 21:56:41.139603: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 9971834001842941242 [worker-1]: 2023-11-24 21:56:41.139961: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:41.166455: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 13185833584839473121 [worker-0]: 2023-11-24 21:56:41.166682: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-11-24 21:56:41.177049: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24071 [worker-0]: 2023-11-24 21:56:41.196287: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 13285026206644634738 [worker-3]: 2023-11-24 21:56:41.197304: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I1124 21:56:41.203906 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1124 21:56:41.209248 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I1124 21:56:41.207535 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1124 21:56:41.207984 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I1124 21:56:41.260956 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1124 21:56:41.261857 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1124 21:56:41.262085 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I1124 21:56:41.259254 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I1124 21:56:41.259779 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:41.260005 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1124 21:56:41.306072 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1124 21:56:41.306591 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1124 21:56:41.306818 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1124 21:56:41.322808 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1124 21:56:41.346523 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1124 21:56:41.346781 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24074', 'localhost:24073', 'localhost:24072', 'localhost:24071']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I1124 21:56:41.460921 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I1124 21:56:41.460923 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:41.461688 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1124 21:56:41.461933 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1124 21:56:41.462246 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 39 [worker-1]: I1124 21:56:41.462450 281473073576608 failure_handler_test.py:197] Start training at 39 [worker-1]: INFO:tensorflow:training restarted [worker-1]: I1124 21:56:41.465047 281473073576608 failure_handler_test.py:207] training restarted [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I1124 21:56:41.476217 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1124 21:56:41.476583 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1124 21:56:41.476926 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 39 [worker-3]: I1124 21:56:41.477132 281473073576608 failure_handler_test.py:197] Start training at 39 [worker-3]: INFO:tensorflow:training restarted [worker-3]: I1124 21:56:41.479566 281473073576608 failure_handler_test.py:207] training restarted [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1124 21:56:41.483209 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:41.489848 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1124 21:56:41.503801 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I1124 21:56:41.504153 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W1124 21:56:41.504486 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 39 [worker-2]: I1124 21:56:41.504692 281473073576608 failure_handler_test.py:197] Start training at 39 [worker-2]: INFO:tensorflow:training restarted [worker-2]: I1124 21:56:41.518435 281473073576608 failure_handler_test.py:207] training restarted [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1124 21:56:41.536211 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1124 21:56:41.536565 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W1124 21:56:41.536905 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 39 [worker-0]: I1124 21:56:41.537114 281473073576608 failure_handler_test.py:197] Start training at 39 [worker-0]: INFO:tensorflow:training restarted [worker-0]: I1124 21:56:41.567573 281473073576608 failure_handler_test.py:207] training restarted [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.729577 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.779170 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.784493 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.794681 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:41.939935 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:41.949578 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:41.983108 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:41.987877 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.088553 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.083672 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.101730 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.116888 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.198925 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.221663 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.209560 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.221643 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.336276 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.347650 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.377762 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.462069 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8410c5e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:42.608258 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff8410c5e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff841085e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:42.610901 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff841085e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.622136 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff841085e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:42.627778 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff841085e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff841045e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:42.628520 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff841045e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.638492 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.635586 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.639087 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84104ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W1124 21:56:42.699501 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff84104ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I1124 21:56:42.699952 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8410cca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W1124 21:56:42.701879 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff8410cca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I1124 21:56:42.702292 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84108ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff84108ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:42.702772 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff84108ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 2 finished [worker-2]: W1124 21:56:42.702805 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff84108ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I1124 21:56:42.703200 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I1124 21:56:42.703233 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.713700 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.716506 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.725198 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.725774 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.839040 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.847131 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.867406 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.882596 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:42.960154 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:42.971215 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:42.972959 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:42.991888 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.091135 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.097446 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.099017 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.131661 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.243485 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.237558 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.254233 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.248043 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.349877 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.351691 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.371803 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.361743 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.490249 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.492021 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.525312 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.527179 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.625852 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.620230 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.626016 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.647209 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.747593 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.753955 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.770768 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.769538 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:43.882382 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:43.882389 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:43.911668 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:43.907109 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.008603 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.016512 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.037705 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.025770 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.137783 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.144333 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.156874 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.155124 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.233938 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.238682 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.251824 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.257829 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.386744 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.373315 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.387249 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.387104 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.478642 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.481314 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.481388 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.484533 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I1124 21:56:44.587858 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I1124 21:56:44.588336 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I1124 21:56:44.596770 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-3]: I1124 21:56:44.576831 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.589387 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.623994 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.662994 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.691529 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.811719 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.812857 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.831557 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.871510 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:44.951380 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:44.947732 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:44.957775 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:44.951697 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.093528 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.105518 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.111834 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.094010 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.217187 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.223932 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.231705 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.244982 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.379202 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.401338 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.407308 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.413995 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.550986 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.552572 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.539723 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.562204 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.672861 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.681590 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.704658 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.768600 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:45.928964 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:45.918552 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:45.937708 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:45.929772 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.023283 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.031623 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.018221 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.041786 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.163541 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.191690 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.201541 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.211963 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.307296 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.310151 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.303273 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.312510 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.466889 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.481826 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.494497 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.518898 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.624274 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.628110 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.634068 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.661668 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.783673 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.791663 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.813235 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.814379 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: I1124 21:56:46.928463 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I1124 21:56:46.938577 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:46.951748 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.936871 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:46.947834 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:46.961545 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.972343 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:46.983551 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.085165 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.096340 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.106373 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.116576 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.257294 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.259952 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.278980 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.281675 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.373923 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.381683 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.411777 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.403345 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.549342 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.560116 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.548379 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.551825 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.655388 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.671669 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.726689 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.801782 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.879108 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.881341 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.886452 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:47.910327 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:47.979015 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:47.969874 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:47.994333 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.004520 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.078240 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.080042 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.093465 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.109237 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.178884 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.180979 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.183027 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.241843 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.338431 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.348875 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.354518 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.372222 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.439157 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.461807 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.477334 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.457596 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.587605 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.621717 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.635209 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.666109 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:48.805440 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:48.806620 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:48.805400 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:48.806771 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.052299 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.059675 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.052600 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.086815 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:epoch 5 finished [worker-3]: INFO:tensorflow:epoch 5 finished [worker-0]: I1124 21:56:49.146099 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-3]: I1124 21:56:49.146096 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-2]: I1124 21:56:49.146424 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:epoch 5 finished [worker-1]: I1124 21:56:49.146401 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.159091 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.160149 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.162392 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.163953 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.236449 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.247485 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.247626 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.238708 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.321628 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.321635 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.320819 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.325772 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.419103 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.422706 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.440448 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.441462 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.646820 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.643799 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.651704 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.651607 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.748776 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.758550 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.768806 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.758465 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.830259 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.831065 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.835060 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.853603 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:49.923388 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:49.922201 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:49.923394 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:49.963129 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.027560 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.033742 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.036737 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.043824 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.105691 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.104676 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.115432 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.147185 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.206955 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.209307 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.217010 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.217059 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.297422 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.304728 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.297413 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.297325 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.364281 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.366739 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.375213 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.367650 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.439296 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.444774 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.446518 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.443624 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:50.560690 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.564637 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.597256 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.612052 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-3]: I1124 21:56:50.926735 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-0]: I1124 21:56:50.928876 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-2]: I1124 21:56:50.946673 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:50.948765 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-3]: I1124 21:56:50.948833 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.966898 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:50.991436 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:50.991431 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:51.162860 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:51.171734 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:51.173098 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:51.227075 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:51.471364 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:51.495170 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:51.537923 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:51.517639 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:51.694900 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:51.691543 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:51.686082 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:51.703008 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:51.842953 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:51.833030 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:51.842997 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:51.851935 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:51.966703 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:51.964348 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:51.982795 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:52.005971 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:52.072678 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:52.082088 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:52.108195 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:52.133555 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:52.222899 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:52.222905 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:52.232884 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:52.232886 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:52.405625 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:52.402382 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:52.432837 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:52.443002 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:52.521772 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:52.540404 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:52.608281 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:52.628509 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:52.765333 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:52.765800 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:52.773108 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:52.799607 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:52.867650 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:52.875127 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:52.894975 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:52.906792 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:52.988713 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:53.002913 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:53.022844 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:53.057950 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:53.175816 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:53.185825 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:53.186333 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:53.188162 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:53.298245 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:53.310739 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:53.332611 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:53.345357 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I1124 21:56:53.428033 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I1124 21:56:53.429206 281473073576608 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-0]: I1124 21:56:53.436989 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I1124 21:56:53.438322 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-2]: I1124 21:56:53.437325 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I1124 21:56:53.439838 281473073576608 failure_handler_test.py:245] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I1124 21:56:53.447212 281473073576608 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I1124 21:56:53.451660 281473073576608 failure_handler_test.py:245] Training finished. I1124 21:56:53.766860 281473177189024 multi_process_runner.py:646] worker-0 exit code: 0 I1124 21:56:53.767208 281473177189024 multi_process_runner.py:646] worker-1 exit code: 0 I1124 21:56:53.767384 281473177189024 multi_process_runner.py:646] worker-2 exit code: 0 I1124 21:56:53.767550 281473177189024 multi_process_runner.py:646] worker-3 exit code: 0 I1124 21:56:53.770211 281473177189024 multi_process_runner.py:662] Joining log reading threads. I1124 21:56:53.770526 281473177189024 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 21.9s I1124 21:56:54.048272 281473177189024 test_util.py:2544] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 21.9s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24062 I1124 21:56:54.052680 281473177189024 test_util.py:3887] Using local port 24062 INFO:tensorflow:Using local port 24061 I1124 21:56:54.054799 281473177189024 test_util.py:3887] Using local port 24061 INFO:tensorflow:Using local port 24060 I1124 21:56:54.056839 281473177189024 test_util.py:3887] Using local port 24060 INFO:tensorflow:Using local port 24059 I1124 21:56:54.058724 281473177189024 test_util.py:3887] Using local port 24059 INFO:tensorflow:Cluster starting. I1124 21:56:54.110278 281473177189024 failure_handler_test.py:297] Cluster starting. [worker-0]: I1124 21:56:54.262721 281473073576608 multi_process_runner.py:840] Subprocess with PID 1359172 (worker, 0) is now being started. [worker-0]: I1124 21:56:54.263194 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24062", "localhost:24061", "localhost:24060", "localhost:24059"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I1124 21:56:54.299146 281473073576608 multi_process_runner.py:840] Subprocess with PID 1359293 (worker, 1) is now being started. [worker-1]: I1124 21:56:54.299596 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24062", "localhost:24061", "localhost:24060", "localhost:24059"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I1124 21:56:54.302930 281473073576608 multi_process_runner.py:840] Subprocess with PID 1359336 (worker, 3) is now being started. [worker-2]: I1124 21:56:54.303231 281473073576608 multi_process_runner.py:840] Subprocess with PID 1359329 (worker, 2) is now being started. [worker-2]: I1124 21:56:54.303622 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24062", "localhost:24061", "localhost:24060", "localhost:24059"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I1124 21:56:54.303377 281473073576608 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24062", "localhost:24061", "localhost:24060", "localhost:24059"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: 2023-11-24 21:56:54.486899: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24059 [worker-2]: 2023-11-24 21:56:54.491760: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24060 [worker-1]: 2023-11-24 21:56:54.531364: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24061 [worker-0]: 2023-11-24 21:56:54.536689: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:24062 [worker-0]: 2023-11-24 21:56:54.576565: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 12719544589303494470 [worker-3]: 2023-11-24 21:56:54.581442: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:54.589762: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 9638513810909793737 [worker-2]: 2023-11-24 21:56:54.590022: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:54.593444: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 876973790764418200 [worker-0]: 2023-11-24 21:56:54.593621: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-11-24 21:56:54.637123: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service.cc:553] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 6293840615376239144 [worker-1]: 2023-11-24 21:56:54.672086: I external/local_tsl/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I1124 21:56:54.679490 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I1124 21:56:54.683217 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I1124 21:56:54.678514 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I1124 21:56:54.692221 281473073576608 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I1124 21:56:54.733546 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I1124 21:56:54.734098 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I1124 21:56:54.734324 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I1124 21:56:54.734421 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I1124 21:56:54.734925 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I1124 21:56:54.735150 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:54.733499 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I1124 21:56:54.735048 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I1124 21:56:54.735303 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I1124 21:56:54.771414 281473073576608 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I1124 21:56:54.771947 281473073576608 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I1124 21:56:54.772171 281473073576608 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24062', 'localhost:24061', 'localhost:24060', 'localhost:24059']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I1124 21:56:54.867967 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I1124 21:56:54.868012 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I1124 21:56:54.877094 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I1124 21:56:54.883474 281473073576608 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I1124 21:56:54.886563 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I1124 21:56:54.886950 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W1124 21:56:54.887316 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I1124 21:56:54.887526 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I1124 21:56:54.896796 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I1124 21:56:54.897263 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W1124 21:56:54.897674 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I1124 21:56:54.897886 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I1124 21:56:54.911630 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I1124 21:56:54.912092 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W1124 21:56:54.912508 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I1124 21:56:54.912724 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I1124 21:56:54.916887 281473073576608 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I1124 21:56:54.917898 281473073576608 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W1124 21:56:54.918246 281473073576608 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I1124 21:56:54.918452 281473073576608 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.110649 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.168494 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.207036 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.289803 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.363476 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.363483 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.373068 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.366273 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I1124 21:56:58.197924 281473177189024 failure_handler_test.py:302] sending sigterm INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 18.06s I1124 21:57:12.104550 281473177189024 test_util.py:2544] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 18.06s [worker-2]: I1124 21:56:55.435348 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.434597 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.497168 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.497054 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker ====================================================================== ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='manager', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process ---------------------------------------------------------------------- Ran 4 tests in 49.827s FAILED (errors=1) [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.557795 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.557814 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840fc550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W1124 21:56:55.606691 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W1124 21:56:55.606421 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840fc550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.619005 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I1124 21:56:55.618842 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W1124 21:56:55.668190 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840fcc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W1124 21:56:55.667953 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840fcc10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: I1124 21:56:55.679668 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.679609 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.739410 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.738818 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.798731 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.798729 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.860085 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.859732 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.919766 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.920454 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.434450 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:55.979284 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:55.980052 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.497321 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.434678 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.038410 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.038375 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.096744 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.557606 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.497292 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.096761 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840f4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.155357 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W1124 21:56:55.606193 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840f4550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I1124 21:56:55.557677 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.154529 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-0]: I1124 21:56:56.202231 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-3]: I1124 21:56:55.618265 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.202415 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-1]: W1124 21:56:55.606487 281473073576608 polymorphic_function.py:157] 5 out of the last 5 calls to .wrapped_fn at 0xffff840f8550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840f4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.213348 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W1124 21:56:55.667712 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840f4c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: I1124 21:56:56.214779 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.616828 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I1124 21:56:56.272674 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.678869 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.272661 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: W1124 21:56:55.668021 281473073576608 polymorphic_function.py:157] 6 out of the last 6 calls to .wrapped_fn at 0xffff840f8c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.336424 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.739071 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.335841 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.679459 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.395745 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.798971 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.396876 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.739285 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.457247 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.859967 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.457849 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.798952 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.518243 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.518237 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.919743 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.860036 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:55.978311 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.581356 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.581340 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.919864 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.037445 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.643282 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:55.979593 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.643807 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.096737 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.038519 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.704906 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.704879 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.152821 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-1]: I1124 21:56:56.096797 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.766790 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.766067 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.202035 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-0]: I1124 21:56:56.824084 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.824059 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.213109 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.881621 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.880637 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.273621 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.938357 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.937552 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.335436 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:56.993949 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.397174 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.050827 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I1124 21:56:56.458409 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.097478 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:56.993304 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.519287 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.050517 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.109480 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.582292 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.164739 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.220772 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.097283 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-3]: I1124 21:56:56.644215 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.279091 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.706685 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.517443 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.766478 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.581912 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.108632 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.154419 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.638413 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.822970 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.880936 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.202354 281473073576608 failure_handler_test.py:195] epoch 0 finished [worker-0]: I1124 21:56:57.165196 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.695980 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.938102 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.213494 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.222302 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.753889 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:56.993263 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.272726 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.279396 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.814208 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.050292 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.334877 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.491097 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-2]: I1124 21:56:57.873524 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.097091 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.395832 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.581900 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.929213 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.457295 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.108651 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.638200 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:57.986277 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.166290 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.518266 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.694211 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.042379 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.581525 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.222634 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.750384 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.098238 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-1]: I1124 21:56:56.643823 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.280148 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.811815 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.143533 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.704952 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.871344 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.451189 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.155974 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.767349 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.581807 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.928415 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.212142 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.824051 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.638980 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:57.984389 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.267962 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.695607 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.880596 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.041517 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.322855 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.753577 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.937642 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.097721 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.378954 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I1124 21:56:57.814015 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:56.993931 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.143236 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.435903 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.873356 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.049782 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.154298 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.492472 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.928932 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.097440 281473073576608 failure_handler_test.py:195] epoch 1 finished [worker-0]: I1124 21:56:58.211050 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.548943 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:57.984898 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.108273 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.266079 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.605033 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.042682 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.164683 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.321763 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.661038 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.097913 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.220865 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.378262 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.716545 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.143163 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-1]: I1124 21:56:57.279024 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.435360 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.771405 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.156478 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.491341 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.492274 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.829681 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.211864 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.582247 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.548120 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.886994 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.638376 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.604184 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.267150 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.322567 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.379684 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.944070 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.659949 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-0]: I1124 21:56:58.715799 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:58.991602 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.771393 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.002257 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.829782 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.060134 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.114793 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.886891 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.168561 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.944156 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-2]: I1124 21:56:59.222429 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:58.991450 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-2]: I1124 21:56:59.275319 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.002094 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.330788 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.059649 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.387523 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.113921 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.444534 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.168025 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.502666 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.222231 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.558433 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.274705 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.612815 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.331576 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.667953 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.388423 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.722517 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.445513 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.777298 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: I1124 21:56:59.503061 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.822750 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.558388 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.832537 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.612917 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.887423 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.667949 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.942466 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.722488 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:56:59.998085 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.777406 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.052407 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.822603 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-2]: I1124 21:57:00.106621 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.832506 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.162220 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.435577 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.887915 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.216742 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.492316 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.942835 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.270724 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.548823 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:56:59.997430 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.324650 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.604821 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.052342 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.379199 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.661358 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.106356 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.433293 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.716328 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.162184 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.487539 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.771214 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.216303 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.541187 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.829512 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.270672 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.595160 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.886595 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.324586 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.943781 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.379092 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:58.991251 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-0]: I1124 21:57:00.433137 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.487411 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.540839 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.594966 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-0]: I1124 21:57:00.640318 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.649885 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.703909 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.001717 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.757545 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.059815 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.813416 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.114479 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.868123 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.169176 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.922063 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.222153 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:00.976550 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.275612 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.030668 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.330458 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.085112 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.387263 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.139275 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.444301 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.194325 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.502484 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.249893 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.558023 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.305181 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.612515 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.358136 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.667593 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.413102 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.722177 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.460689 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-3]: I1124 21:56:59.777009 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: I1124 21:57:01.472979 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.822407 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.530130 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.832105 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.587371 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.887259 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.644113 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.942181 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.701142 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:56:59.998545 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.758085 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.052066 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.815396 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.106191 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.872915 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.161931 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.930585 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.215966 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:01.985040 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.270444 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.044022 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.324432 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.104598 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.378800 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.163317 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.433069 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.332942 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.487286 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.695772 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.444883 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.640495 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-3]: I1124 21:57:00.540870 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.750927 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.489195 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-2]: I1124 21:57:00.650328 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.594901 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.811915 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I1124 21:57:02.490416 281473073576608 failure_handler_test.py:245] Training finished. [worker-2]: I1124 21:57:00.704190 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.640140 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-1]: I1124 21:56:57.871530 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.757959 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.650034 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.928767 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.813501 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:57.985419 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.703859 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.867959 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.922052 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.757760 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:00.977044 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.813414 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.030685 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.867577 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.085141 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.922348 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.140025 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:00.976848 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.194792 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.030467 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.249215 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.084879 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.305460 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.139133 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.358909 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.194535 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.412050 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.249008 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.460826 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-3]: I1124 21:57:01.305304 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.472211 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.358640 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.529382 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.411746 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.586297 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.460437 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-2]: I1124 21:57:01.643259 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.471759 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.700239 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.529090 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.757306 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.585983 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.814555 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.643477 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.872191 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.699945 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.929955 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.757012 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:01.986014 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.814268 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.045814 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.871915 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.106222 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.929641 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.164386 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:01.985817 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.386301 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:02.045814 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.444669 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-3]: I1124 21:57:02.106226 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.489370 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-3]: I1124 21:57:02.163881 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I1124 21:57:02.490739 281473073576608 failure_handler_test.py:245] Training finished. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:02.389622 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I1124 21:57:02.444474 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I1124 21:57:02.489039 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I1124 21:57:02.489945 281473073576608 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.041748 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.097748 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I1124 21:56:58.143432 281473073576608 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.154807 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.211325 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.266303 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.321927 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.378324 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.436692 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.492652 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.548129 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.604858 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.659975 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.715840 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.771409 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.829820 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.886912 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:58.944175 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I1124 21:56:58.991580 281473073576608 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.002137 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.059780 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.114382 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.168097 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.221931 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.274874 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.331609 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.388453 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.445541 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.503131 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.558421 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.612931 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.668122 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.722496 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.777441 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I1124 21:56:59.822726 281473073576608 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.833134 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.887901 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.942677 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:56:59.997441 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.052492 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.106412 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.162339 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.216316 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.270685 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.324589 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.379130 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.433150 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.487468 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.540984 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.595070 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-1]: I1124 21:57:00.640432 281473073576608 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.650649 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.703986 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.758016 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.813617 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.867862 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.922118 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:00.976654 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.030786 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.085297 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.139413 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.194346 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.249884 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.305246 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.358362 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.413130 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I1124 21:57:01.460853 281473073576608 failure_handler_test.py:195] epoch 6 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.473833 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.530186 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.587350 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.644131 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.701170 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.758075 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.815417 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.872928 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.930690 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:01.985700 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:02.044016 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:02.104602 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:02.163428 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:02.332590 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I1124 21:57:02.444950 281473073576608 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I1124 21:57:02.489342 281473073576608 failure_handler_test.py:195] epoch 7 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I1124 21:57:02.490568 281473073576608 failure_handler_test.py:245] Training finished. ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 44.6s //tensorflow/c:c_api_function_test PASSED in 28.9s //tensorflow/c:c_api_test_cpu PASSED in 37.6s //tensorflow/c:c_test PASSED in 34.2s //tensorflow/c:env_test_cpu PASSED in 31.8s //tensorflow/c:kernels_test_cpu PASSED in 40.7s //tensorflow/c:ops_test PASSED in 34.8s //tensorflow/c:tf_status_helper_test PASSED in 0.1s //tensorflow/c:while_loop_test PASSED in 39.3s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 34.7s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 32.5s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 37.9s //tensorflow/c/eager:c_api_test_cpu PASSED in 37.5s //tensorflow/c/eager:custom_device_test PASSED in 40.2s //tensorflow/c/eager:dlpack_test_cpu PASSED in 36.9s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 36.2s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 38.5s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 38.7s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.6s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.7s //tensorflow/c/experimental/grappler:grappler_test PASSED in 34.2s //tensorflow/c/experimental/next_pluggable_device:tensor_pjrt_buffer_util_test PASSED in 9.0s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 1.3s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.7s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.8s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 16.5s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 15.3s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 38.6s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 14.4s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 17.2s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 18.7s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 17.6s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 39.4s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.1s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.6s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 0.6s //tensorflow/c/kernels:summary_op_test PASSED in 0.8s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.1s //tensorflow/cc:cc_op_gen_test PASSED in 0.7s //tensorflow/cc:client_client_session_test PASSED in 2.2s //tensorflow/cc:coordinator_test PASSED in 4.6s //tensorflow/cc:framework_cc_ops_test PASSED in 1.7s //tensorflow/cc:framework_gradient_checker_test PASSED in 2.2s //tensorflow/cc:framework_gradients_test PASSED in 4.4s //tensorflow/cc:framework_scope_test PASSED in 1.0s //tensorflow/cc:framework_while_gradients_test PASSED in 3.0s //tensorflow/cc:gradients_array_grad_test PASSED in 4.3s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 1.9s //tensorflow/cc:gradients_functional_grad_test PASSED in 1.7s //tensorflow/cc:gradients_image_grad_test PASSED in 5.2s //tensorflow/cc:gradients_linalg_grad_test PASSED in 2.1s //tensorflow/cc:gradients_manip_grad_test PASSED in 1.8s //tensorflow/cc:gradients_math_grad_test PASSED in 9.6s //tensorflow/cc:gradients_nn_grad_test PASSED in 3.3s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 1.7s //tensorflow/cc:ops_const_op_test PASSED in 0.7s //tensorflow/cc:ops_while_loop_test PASSED in 1.7s //tensorflow/cc:queue_runner_test PASSED in 12.6s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.1s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 36.6s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.1s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.3s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 35.5s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 40.4s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 36.2s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.1s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.1s //tensorflow/cc/saved_model:fingerprinting_chunked_test PASSED in 0.4s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.0s //tensorflow/cc/saved_model:fingerprinting_utils_test PASSED in 0.2s //tensorflow/cc/saved_model:metrics_test PASSED in 0.4s //tensorflow/cc/saved_model:reader_test PASSED in 0.6s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 26.5s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 6.1s //tensorflow/cc/saved_model:util_test PASSED in 0.2s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 40.2s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 1.8s //tensorflow/compiler/aot:codegen_test PASSED in 35.9s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 22.3s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 14.5s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 6.8s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 7.1s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 24.2s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 5.7s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 28.4s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 28.9s //tensorflow/compiler/jit:device_util_test PASSED in 6.5s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 0.8s //tensorflow/compiler/jit:node_matchers_test PASSED in 0.5s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 12.5s //tensorflow/compiler/jit:shape_inference_test PASSED in 0.4s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 25.3s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 8.6s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 4.7s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 7.7s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 28.9s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 29.1s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 14.3s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 29.1s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite:offset_buffer_test PASSED in 0.1s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.3s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 6.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 0.6s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.4s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 14.6s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 16.3s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:call_xla_module_to_stablehlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:compose-uniform-quantized-type.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize_hlo.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tfl_legalize_hlo.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:tfl_legalize_hlo_custom_call.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfold_splat_constant_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:uniform-quantized-stablehlo-to-tfl.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tensorlist.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/tests:optimize_batch_matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 3.2s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:reduce-type-precision.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 16.3s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants_offset.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op_offset.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:mix_tflite_stablehlo.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_const.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_custom_call.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:variable.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_tensorlist_reserve.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:u16_quant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variable.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_types_test PASSED in 17.3s //tensorflow/compiler/mlir/quantization/stablehlo:math_utils_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo:stablehlo_type_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/stablehlo:tf_type_utils_test PASSED in 19.8s //tensorflow/compiler/mlir/quantization/stablehlo:uniform_quantized_types_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo/python:quantize_model_test PASSED in 75.5s //tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/stablehlo/tests:stablehlo_op_quant_spec_test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibration_algorithm_test PASSED in 40.2s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibration_statistics_collector_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 23.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/cc:constant_fold_test PASSED in 3.3s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 5.4s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/ops:tf_op_quant_spec_test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/ops:tf_quantize_op_test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 54.6s //tensorflow/compiler/mlir/quantization/tensorflow/python:py_function_lib_py_test PASSED in 22.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 22.2s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 13.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:add_dump_tensor_op.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:add_quantization_unit_loc.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_xla_op_to_tf_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_hashtable_ops_as_args.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla_selective_quantization.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_duplicate_resource_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 4.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:propagate_quantize_type.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composit_functions_debugging.mlir.test PASSED in 6.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 19.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 4.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_weights.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 17.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_uniform_attribute_utils_test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 41.7s //tensorflow/compiler/mlir/stablehlo:stablehlo_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 7.0s //tensorflow/compiler/mlir/tensorflow:call_graph_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:data_dumper_logger_config_test PASSED in 6.8s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 16.3s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:tf_mlir_translate_registration_test PASSED in 21.6s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:xla_rewrite_util_test PASSED in 10.3s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:colocate_tpu_copy_with_dynamic_shape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:embedding_program_key.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:embedding_sequencing.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:move_tpu_compile_to_front.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:print.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_asset_sinking.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tfrt_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 13.9s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:verify_for_export.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_sharding_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow/tests:xla_validate_iputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:convert_mhlo_quant_to_int.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 17.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 4.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graph:convert_tensor.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/transforms:verify_no_outside_compilation_markers_pass_test PASSED in 22.9s //tensorflow/compiler/mlir/tensorflow/transforms/host_runtime:lower_cluster_to_runtime_ops_test PASSED in 17.4s //tensorflow/compiler/mlir/tf2xla/api/v1:cluster_tf_test PASSED in 31.9s //tensorflow/compiler/mlir/tf2xla/api/v1:compile_mlir_util_test PASSED in 5.8s //tensorflow/compiler/mlir/tf2xla/api/v1:compile_tf_graph_test PASSED in 0.2s //tensorflow/compiler/mlir/tf2xla/api/v1:tf_dialect_to_executor_test PASSED in 18.3s //tensorflow/compiler/mlir/tf2xla/api/v2:cluster_tf_test PASSED in 38.3s //tensorflow/compiler/mlir/tf2xla/api/v2:legalize_tf_test PASSED in 29.8s //tensorflow/compiler/mlir/tf2xla/api/v2:tf_dialect_to_executor_test PASSED in 17.5s //tensorflow/compiler/mlir/tf2xla/internal:clustering_bridge_passes_test PASSED in 6.7s //tensorflow/compiler/mlir/tf2xla/internal:compilation_timer_test PASSED in 0.3s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_mlir_test PASSED in 26.6s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_to_hlo_test PASSED in 29.2s //tensorflow/compiler/mlir/tf2xla/internal:logging_hooks_test PASSED in 17.8s //tensorflow/compiler/mlir/tf2xla/internal:mlir_pass_instrumentation_test PASSED in 9.3s //tensorflow/compiler/mlir/tf2xla/internal:test_matchers_test PASSED in 6.9s //tensorflow/compiler/mlir/tf2xla/internal/inference:inference_metrics_pass_test PASSED in 19.0s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_clustering_pass_test PASSED in 16.4s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_clustering_pass_test.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/internal/passes:verify_input_dialect_to_executor_pass_test.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/internal/utils:dialect_detection_utils_test PASSED in 0.5s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 17.5s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-quant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 10.4s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/transforms:legalization_op_config_test PASSED in 39.1s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 20.2s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 22.1s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 4.1s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 14.5s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 12.7s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 21.9s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 7.3s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 31.2s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 27.8s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 27.4s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 31.1s //tensorflow/compiler/mlir/tfrt/tests:batch_function_fallback_resource_variable_as_captured_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:batch_function_lowering.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:convert_ref_variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:cross_device_transfer.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:deduplicate_if_results.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:fuse_tpu_compile_and_execute_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops_mlrt.mlir.test PASSED in 17.4s //tensorflow/compiler/mlir/tfrt/tests:optimize.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tfrt/tests:remove_device_attribute.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tfrt/tests:runtime_lowering_gpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:runtime_lowering_tpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:sink_in_invariant_ops.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_fallback.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_lowering.mlir.test PASSED in 3.4s //tensorflow/compiler/mlir/tfrt/tests:xla_rewrite.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tfrt/tests/analysis:cost_analysis.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/analysis:tensor_array_side_effect_analysis.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/analysis:update_op_cost_in_tfrt_mlir_test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/ifrt:rewrite_cluster_to_ifrt_call.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/ir:fallback_opt.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/ir:tfrt_fallback_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tfrt/tests/mlrt:assign_op_key.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/mlrt:async_while.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/mlrt:fuse_mlrt_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:inline.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/mlrt:parallelization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tf_to_mlrt.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tpu_conversions.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/mlrt:while_to_map_fn.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:basic.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate_failed.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:const_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:control_flow.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:decompose_resource_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:derived_attrs.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:device_conversion.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:errors.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_canonicalization.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_inline.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes_multiple_callers.mlir.test PASSED in 16.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_use_fallback_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:insert_fallback_tensor_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:merge_tf_if_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:optimize_tf_control_flow_side_effect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:remove_tf_if_const_args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:reorder_assert.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:side_effects.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline_refvar.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:whileop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/translate/mlrt:mlir_to_bytecode_test PASSED in 0.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:func_to_jit_invocations.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 6.9s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-stateful.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 1.8s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 29.3s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 35.6s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 13.3s //tensorflow/compiler/tests:adam_test_cpu PASSED in 15.2s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 11.7s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 22.7s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 19.5s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 9.3s //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 10.8s //tensorflow/compiler/tests:case_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 11.1s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 19.1s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 15.0s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 19.7s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 42.6s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 11.9s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 13.0s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 16.4s //tensorflow/compiler/tests:cond_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:const_test_cpu PASSED in 13.0s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 17.4s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 17.8s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 17.1s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 15.2s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 13.9s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 9.6s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 10.7s //tensorflow/compiler/tests:eager_test_cpu PASSED in 40.5s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 11.0s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 13.5s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 15.5s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 13.4s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 16.2s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 17.7s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 13.9s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 31.1s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 14.3s //tensorflow/compiler/tests:function_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 10.6s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 13.7s //tensorflow/compiler/tests:gather_test_cpu PASSED in 47.8s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 63.1s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 12.2s //tensorflow/compiler/tests:jit_test_cpu PASSED in 51.0s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 36.0s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 12.8s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 12.3s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 44.5s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 15.1s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 17.3s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 41.1s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 41.7s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 21.0s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 22.8s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 10.9s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 14.9s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 14.0s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 13.9s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 11.5s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 16.7s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 10.7s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 27.9s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 12.4s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 12.2s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 10.4s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 19.3s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 11.7s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 13.9s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 16.1s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 15.5s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 11.9s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 14.5s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 24.5s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 25.3s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 12.6s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 12.6s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 26.2s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 30.6s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 24.4s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 18.6s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 37.6s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 26.3s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 12.4s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 10.6s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 15.2s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 40.3s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 12.5s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 19.1s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 20.0s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 18.5s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 18.3s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 9.3s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 28.0s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 26.5s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:while_test_cpu PASSED in 21.7s //tensorflow/compiler/tests:xla_call_module_no_platform_check_test_cpu PASSED in 16.4s //tensorflow/compiler/tests:xla_call_module_no_shape_assertions_check_test_cpu PASSED in 12.8s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 31.0s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 9.0s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 19.1s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 21.8s //tensorflow/compiler/tests:xla_dump_to_test_cpu PASSED in 13.0s //tensorflow/compiler/tests:xla_dump_to_test_cpu_mlir_bridge_test PASSED in 10.2s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 39.1s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 42.4s //tensorflow/compiler/tests:xla_test_test PASSED in 10.0s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 7.3s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.2s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 1.3s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 1.6s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 28.1s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 7.3s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 0.5s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 6.4s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 2.0s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.9s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 10.3s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 22.4s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.9s //tensorflow/compiler/tf2xla:type_util_test PASSED in 0.4s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 22.6s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 18.6s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 6.2s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 1.3s //tensorflow/core:@local_tsl__tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.5s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 8.5s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 44.5s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.1s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.9s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.2s //tensorflow/core/api_def:api_test PASSED in 3.5s //tensorflow/core/api_def:update_api_def_test PASSED in 0.1s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 0.6s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.8s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 0.7s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 1.2s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 5.2s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 1.1s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.4s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 3.0s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.2s //tensorflow/core/common_runtime:device_mgr_test PASSED in 0.8s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.6s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 1.3s //tensorflow/core/common_runtime:device_set_test PASSED in 0.8s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 4.5s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 2.6s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.2s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 1.1s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.8s //tensorflow/core/common_runtime:executor_test PASSED in 2.0s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 1.0s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 0.7s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 1.0s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.0s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 2.2s //tensorflow/core/common_runtime:graph_runner_test PASSED in 0.8s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 3.0s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.4s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 0.5s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 1.1s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 2.2s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 2.0s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 2.1s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 3.3s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 1.7s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.1s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.2s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.1s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.3s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 0.8s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 6.1s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.8s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 15.3s //tensorflow/core/common_runtime:pending_counts_test PASSED in 1.0s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 4.9s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 1.0s //tensorflow/core/common_runtime:placer_test PASSED in 1.5s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 1.4s //tensorflow/core/common_runtime:process_util_test PASSED in 0.2s //tensorflow/core/common_runtime:quantize_training_test PASSED in 1.9s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.2s //tensorflow/core/common_runtime:replicate_constants_pass_test PASSED in 0.8s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 1.0s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.3s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.1s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 2.3s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 5.8s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 5.4s //tensorflow/core/common_runtime:session_test PASSED in 0.8s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 0.6s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 0.9s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 0.9s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.1s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 25.5s //tensorflow/core/common_runtime/eager:context_test PASSED in 14.5s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 15.7s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 12.2s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 0.9s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 17.8s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 16.3s //tensorflow/core/common_runtime/eager:execute_test PASSED in 25.3s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 1.0s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 13.8s //tensorflow/core/common_runtime/eager:placement_test PASSED in 10.7s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 13.2s //tensorflow/core/common_runtime/eager:summary_optimizer_test PASSED in 0.1s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 11.3s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 13.0s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.1s //tensorflow/core/common_runtime/gpu:gpu_serving_device_selector_test PASSED in 0.2s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 29.5s //tensorflow/core/common_runtime/next_pluggable_device/c:tf_rendezvous_c_api_test PASSED in 0.2s //tensorflow/core/config:flags_py_test PASSED in 9.2s //tensorflow/core/config:flags_test PASSED in 0.3s //tensorflow/core/data:compression_utils_test PASSED in 1.8s //tensorflow/core/data:dataset_utils_test PASSED in 1.2s //tensorflow/core/data:hash_utils_test PASSED in 1.1s //tensorflow/core/data:metric_utils_test PASSED in 5.7s //tensorflow/core/data:name_utils_test PASSED in 0.3s //tensorflow/core/data:rewrite_utils_test PASSED in 0.5s //tensorflow/core/data:serialization_utils_test PASSED in 0.7s //tensorflow/core/data:snapshot_utils_test PASSED in 0.7s //tensorflow/core/data:split_utils_test PASSED in 0.5s //tensorflow/core/data:standalone_save_restore_test PASSED in 17.4s //tensorflow/core/data:standalone_test PASSED in 4.7s //tensorflow/core/data:tfdataz_metrics_test PASSED in 1.6s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.9s //tensorflow/core/data/service:auto_scaler_test PASSED in 0.1s //tensorflow/core/data/service:common_test PASSED in 0.6s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.8s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.3s //tensorflow/core/data/service:data_service_test PASSED in 14.8s //tensorflow/core/data/service:data_transfer_test PASSED in 0.5s //tensorflow/core/data/service:dataset_store_test PASSED in 0.9s //tensorflow/core/data/service:dispatcher_client_test PASSED in 4.0s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.4s //tensorflow/core/data/service:graph_rewriters_test PASSED in 0.6s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 3.9s //tensorflow/core/data/service:grpc_util_test PASSED in 0.7s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 2.2s //tensorflow/core/data/service:journal_test PASSED in 0.7s //tensorflow/core/data/service:logging_utils_test PASSED in 0.4s //tensorflow/core/data/service:task_runner_test PASSED in 3.0s //tensorflow/core/data/service:test_util_test PASSED in 1.5s //tensorflow/core/data/service:url_test PASSED in 0.5s //tensorflow/core/data/service:utils_test PASSED in 0.7s //tensorflow/core/data/service:validate_utils_test PASSED in 0.1s //tensorflow/core/data/service:worker_client_test PASSED in 2.4s //tensorflow/core/data/service:worker_impl_test PASSED in 2.1s //tensorflow/core/data/service/client:data_service_client_test PASSED in 2.9s //tensorflow/core/data/service/client:utils_test PASSED in 2.5s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.5s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 17.4s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.6s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.1s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 2.0s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 0.7s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 5.1s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 2.8s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.1s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.4s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.7s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 4.4s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 1.0s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.6s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 0.5s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.5s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.5s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.2s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.2s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.7s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.5s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 0.7s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.2s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.7s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 22.9s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 14.2s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 55.2s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 46.3s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 34.9s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 2.9s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 0.7s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 0.7s //tensorflow/core/example:example_parser_configuration_test PASSED in 0.9s //tensorflow/core/example:feature_util_test PASSED in 0.1s //tensorflow/core/framework:allocator_test PASSED in 4.2s //tensorflow/core/framework:attr_value_util_test PASSED in 1.1s //tensorflow/core/framework:batch_util_test PASSED in 0.9s //tensorflow/core/framework:bfloat16_test PASSED in 1.1s //tensorflow/core/framework:common_shape_fns_test PASSED in 1.0s //tensorflow/core/framework:dataset_test PASSED in 0.8s //tensorflow/core/framework:device_base_test PASSED in 0.8s //tensorflow/core/framework:disable_jit_test PASSED in 1.4s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.1s //tensorflow/core/framework:framework_op_segment_test PASSED in 1.0s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.1s //tensorflow/core/framework:framework_run_handler_test PASSED in 1.7s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.1s //tensorflow/core/framework:full_type_inference_util_test PASSED in 0.8s //tensorflow/core/framework:full_type_util_test PASSED in 1.3s //tensorflow/core/framework:function_test PASSED in 1.0s //tensorflow/core/framework:graph_def_util_test PASSED in 0.8s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 1.3s //tensorflow/core/framework:kernel_def_builder_test PASSED in 1.5s //tensorflow/core/framework:kernel_def_util_test PASSED in 0.8s //tensorflow/core/framework:memory_types_test PASSED in 0.9s //tensorflow/core/framework:model_test PASSED in 0.9s //tensorflow/core/framework:node_def_builder_test PASSED in 2.7s //tensorflow/core/framework:node_def_util_test PASSED in 0.9s //tensorflow/core/framework:node_properties_test PASSED in 0.9s //tensorflow/core/framework:op_compatibility_test PASSED in 0.9s //tensorflow/core/framework:op_def_builder_test PASSED in 0.8s //tensorflow/core/framework:op_def_util_test PASSED in 0.9s //tensorflow/core/framework:op_kernel_test PASSED in 0.9s //tensorflow/core/framework:op_registration_test PASSED in 0.8s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 1.0s //tensorflow/core/framework:rendezvous_test PASSED in 3.4s //tensorflow/core/framework:resource_handle_test PASSED in 0.2s //tensorflow/core/framework:resource_mgr_test PASSED in 1.7s //tensorflow/core/framework:resource_op_kernel_test PASSED in 0.9s //tensorflow/core/framework:shape_inference_test PASSED in 1.4s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 0.9s //tensorflow/core/framework:tensor_matcher_test PASSED in 0.8s //tensorflow/core/framework:tensor_shape_test PASSED in 8.0s //tensorflow/core/framework:tensor_slice_test PASSED in 1.1s //tensorflow/core/framework:tensor_test PASSED in 37.4s //tensorflow/core/framework:tensor_testutil_test PASSED in 1.5s //tensorflow/core/framework:tensor_util_test PASSED in 0.9s //tensorflow/core/framework:tracking_allocator_test PASSED in 0.9s //tensorflow/core/framework:types_test PASSED in 0.9s //tensorflow/core/framework:variant_op_registry_test PASSED in 19.6s //tensorflow/core/framework:variant_test PASSED in 1.1s //tensorflow/core/framework/registration:registration_test PASSED in 0.4s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 9.8s //tensorflow/core/function/capture:capture_container_test PASSED in 9.9s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 20.9s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 20.7s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 9.4s //tensorflow/core/function/polymorphism:function_type_test PASSED in 8.6s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 10.1s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 48.7s //tensorflow/core/function/trace_type:custom_nest_trace_type_test PASSED in 9.4s //tensorflow/core/function/trace_type:default_types_test PASSED in 9.3s //tensorflow/core/function/trace_type:serialization_test PASSED in 9.0s //tensorflow/core/function/trace_type:trace_type_test PASSED in 13.9s //tensorflow/core/graph:algorithm_test PASSED in 1.1s //tensorflow/core/graph:collective_order_test PASSED in 0.6s //tensorflow/core/graph:control_flow_test PASSED in 1.9s //tensorflow/core/graph:costmodel_test PASSED in 0.9s //tensorflow/core/graph:edgeset_test PASSED in 0.8s //tensorflow/core/graph:graph_debug_info_builder_test PASSED in 1.0s //tensorflow/core/graph:graph_def_builder_test PASSED in 0.8s //tensorflow/core/graph:graph_partition_test PASSED in 1.5s //tensorflow/core/graph:graph_test PASSED in 0.9s //tensorflow/core/graph:node_builder_test PASSED in 0.8s //tensorflow/core/graph:optimizer_cse_test PASSED in 1.1s //tensorflow/core/graph:subgraph_test PASSED in 1.0s //tensorflow/core/graph:tensor_id_test PASSED in 0.9s //tensorflow/core/graph:validate_test PASSED in 0.8s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.4s //tensorflow/core/graph/regularization:util_test PASSED in 0.1s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.2s //tensorflow/core/grappler:graph_view_test PASSED in 1.5s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 1.2s //tensorflow/core/grappler:grappler_item_test PASSED in 1.7s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 1.3s //tensorflow/core/grappler:utils_test PASSED in 3.0s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 22.9s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 1.1s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 2.4s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.1s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.2s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 3.3s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.2s //tensorflow/core/grappler/costs:utils_test PASSED in 1.2s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.5s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.9s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 1.8s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 1.7s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 1.4s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 2.7s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 1.9s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 3.6s //tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 2.6s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 5.5s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 1.5s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 3.6s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.3s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 2.4s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 8.3s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 1.8s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:remapper_test_cpu PASSED in 3.2s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 1.9s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 1.2s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 1.0s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:inject_io_prefetch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 3.4s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:remove_compression_map_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.2s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 1.2s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.6s //tensorflow/core/grappler/utils:frame_test PASSED in 0.3s //tensorflow/core/grappler/utils:functions_test PASSED in 1.4s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.4s //tensorflow/core/grappler/utils:graph_view_test PASSED in 1.8s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 5.6s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.6s //tensorflow/core/grappler/utils:scc_test PASSED in 1.2s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.8s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.5s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.1s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.5s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.5s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.0s //tensorflow/core/ir:interfaces_test PASSED in 0.2s //tensorflow/core/ir:ops_test PASSED in 0.2s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.4s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.3s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 2.8s //tensorflow/core/ir:utility_test PASSED in 0.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 3.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 3.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 17.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.7s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 1.4s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 1.2s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 2.2s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.5s //tensorflow/core/ir/types:dialect_test PASSED in 0.3s //tensorflow/core/kernels:as_string_op_test PASSED in 0.7s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.6s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 0.8s //tensorflow/core/kernels:batch_kernels_test PASSED in 43.6s //tensorflow/core/kernels:bias_op_test PASSED in 0.6s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 1.4s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 0.5s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.7s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 1.1s //tensorflow/core/kernels:concat_op_test PASSED in 0.7s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:control_flow_ops_test PASSED in 6.8s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 6.9s //tensorflow/core/kernels:count_ops_test PASSED in 0.9s //tensorflow/core/kernels:cross_op_test PASSED in 0.6s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 1.2s //tensorflow/core/kernels:debug_ops_test PASSED in 0.8s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.5s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.6s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.6s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.3s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.1s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 1.1s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.1s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.4s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.1s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.4s //tensorflow/core/kernels:encode_wav_op_test PASSED in 1.8s //tensorflow/core/kernels:fingerprint_op_test PASSED in 0.7s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 0.6s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.6s //tensorflow/core/kernels:identity_op_test PASSED in 0.7s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 0.8s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.4s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 0.7s //tensorflow/core/kernels:logging_ops_test PASSED in 1.9s //tensorflow/core/kernels:lookup_ops_test PASSED in 0.5s //tensorflow/core/kernels:loss_test PASSED in 0.2s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 3.6s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 0.6s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.2s //tensorflow/core/kernels:mfcc_test PASSED in 0.1s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.4s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.6s //tensorflow/core/kernels:ops_util_test PASSED in 0.5s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:parse_tensor_test PASSED in 0.6s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.6s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.5s //tensorflow/core/kernels:quantize_op_test PASSED in 0.9s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.9s //tensorflow/core/kernels:quantized_add_op_test PASSED in 0.9s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 0.8s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.2s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 1.0s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 2.2s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 0.9s //tensorflow/core/kernels:ragged_range_op_test PASSED in 0.6s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.6s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 0.5s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 0.7s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.7s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:range_sampler_test PASSED in 0.6s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.9s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.7s //tensorflow/core/kernels:requantization_range_op_test PASSED in 0.7s //tensorflow/core/kernels:requantize_op_test PASSED in 1.6s //tensorflow/core/kernels:resource_ops_test PASSED in 0.9s //tensorflow/core/kernels:restore_op_test PASSED in 0.7s //tensorflow/core/kernels:restore_v2_op_test PASSED in 0.8s //tensorflow/core/kernels:reverse_op_test PASSED in 0.8s //tensorflow/core/kernels:roll_op_test PASSED in 0.5s //tensorflow/core/kernels:save_op_test PASSED in 2.5s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.6s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:scatter_op_test PASSED in 0.7s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 8.6s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.2s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 0.4s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.5s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.4s //tensorflow/core/kernels:shape_ops_test PASSED in 1.0s //tensorflow/core/kernels:slice_op_test PASSED in 0.4s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.9s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.9s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 0.7s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.5s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 1.9s //tensorflow/core/kernels:spectrogram_test PASSED in 0.5s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:strided_slice_op_test PASSED in 1.8s //tensorflow/core/kernels:string_format_op_test PASSED in 0.5s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 0.8s //tensorflow/core/kernels:string_split_op_test PASSED in 0.5s //tensorflow/core/kernels:substr_op_test PASSED in 0.5s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_image_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.8s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.8s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.5s //tensorflow/core/kernels:tensor_map_test PASSED in 0.1s //tensorflow/core/kernels:training_ops_test PASSED in 0.5s //tensorflow/core/kernels:transpose_util_test PASSED in 0.6s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 1.7s //tensorflow/core/kernels:unique_op_test PASSED in 1.2s //tensorflow/core/kernels:variable_ops_test PASSED in 1.4s //tensorflow/core/kernels:while_op_test PASSED in 1.0s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.5s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.5s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.9s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 2.2s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 2.4s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 4.1s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.1s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 1.5s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:get_options_op_test PASSED in 0.7s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 0.6s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.7s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 1.8s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.5s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 2.2s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 4.0s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.7s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 0.7s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 0.5s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.8s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.4s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 1.3s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.4s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 0.6s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.2s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 3.9s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 2.5s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 0.6s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 1.5s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.2s //tensorflow/core/kernels/sparse:kernels_test PASSED in 2.6s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 3.9s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 0.5s //tensorflow/core/lib/db:sqlite_test PASSED in 0.2s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 1.2s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.5s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 3.8s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.2s //tensorflow/core/ops:ops_math_grad_test PASSED in 8.1s //tensorflow/core/ops:ops_tests PASSED in 0.7s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.5s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.1s //tensorflow/core/platform:env_test PASSED in 2.5s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 0.3s //tensorflow/core/platform:platform_strings_test PASSED in 0.1s //tensorflow/core/platform:ram_file_system_test PASSED in 12.0s //tensorflow/core/platform:resource_loader_test PASSED in 0.1s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.5s //tensorflow/core/platform:vmodule_test PASSED in 0.2s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.3s //tensorflow/core/profiler/convert:dcn_analysis_test PASSED in 0.1s //tensorflow/core/profiler/convert:dcn_utils_test PASSED in 0.1s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_combiner_test PASSED in 0.4s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.6s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.7s //tensorflow/core/profiler/convert:repository_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_dcn_collective_stats_test PASSED in 0.5s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 16.9s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.5s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 1.1s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 0.6s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 1.3s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 1.2s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 0.5s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 0.5s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.2s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.2s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.1s //tensorflow/core/runtime_fallback/util:type_util_test PASSED in 0.2s //tensorflow/core/summary:schema_test PASSED in 0.1s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.2s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.1s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 10.0s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 6.6s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 5.5s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.2s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.6s //tensorflow/core/tfrt/graph_executor:config_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/attribute:attribute_test PASSED in 0.4s //tensorflow/core/tfrt/mlrt/bytecode:bytecode_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:executable_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:function_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:kernel_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:span_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:context_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:future_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:interpreter_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:register_span_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:value_test PASSED in 0.1s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_concurrent_work_queue_test PASSED in 0.7s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_test PASSED in 0.9s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_util_test PASSED in 0.1s //tensorflow/core/tfrt/runtime:tf_threadpool_concurrent_work_queue_test PASSED in 0.3s //tensorflow/core/tfrt/runtime:work_queue_interface_test PASSED in 0.1s //tensorflow/core/tfrt/utils:graph_partition_test PASSED in 2.0s //tensorflow/core/transforms:eval_utils_test PASSED in 1.3s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.2s //tensorflow/core/util:bcast_test PASSED in 1.0s //tensorflow/core/util:command_line_flags_test PASSED in 1.0s //tensorflow/core/util:debug_data_dumper_test PASSED in 0.9s //tensorflow/core/util:debug_events_writer_test PASSED in 0.4s //tensorflow/core/util:dump_graph_test PASSED in 0.9s //tensorflow/core/util:equal_graph_def_test PASSED in 1.4s //tensorflow/core/util:events_writer_test PASSED in 3.0s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.2s //tensorflow/core/util:example_proto_helper_test PASSED in 0.8s //tensorflow/core/util:exec_on_stall_test PASSED in 5.0s //tensorflow/core/util:fake_clock_env_test PASSED in 2.2s //tensorflow/core/util:incremental_barrier_test PASSED in 0.1s //tensorflow/core/util:matmul_bcast_test PASSED in 1.0s //tensorflow/core/util:memmapped_file_system_test PASSED in 1.1s //tensorflow/core/util:mkl_heuristics_test PASSED in 0.1s //tensorflow/core/util:overflow_test PASSED in 0.2s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 2.2s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.7s //tensorflow/core/util:reffed_status_callback_test PASSED in 0.8s //tensorflow/core/util:reporter_test PASSED in 1.4s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.8s //tensorflow/core/util:semver_test PASSED in 0.7s //tensorflow/core/util:stat_summarizer_test PASSED in 1.1s //tensorflow/core/util:strided_slice_op_test PASSED in 1.4s //tensorflow/core/util:tensor_format_test PASSED in 0.9s //tensorflow/core/util:tensor_slice_reader_test PASSED in 1.8s //tensorflow/core/util:tensor_slice_set_test PASSED in 0.7s //tensorflow/core/util:tensor_slice_util_test PASSED in 0.8s //tensorflow/core/util:tensor_slice_writer_test PASSED in 1.6s //tensorflow/core/util:work_sharder_test PASSED in 2.0s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.1s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.6s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.6s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.2s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.1s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 32.1s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.3s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:decompose_controlflow.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_collective_type_lowering.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 2.2s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_fft.mlir.test PASSED in 2.0s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 1.3s //tensorflow/dtensor/python/tests:api_test PASSED in 30.1s //tensorflow/dtensor/python/tests:array_ops_test_cpu PASSED in 21.9s //tensorflow/dtensor/python/tests:cache_test_cpu PASSED in 18.4s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 21.1s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 24.2s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 8.3s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 65.2s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 22.2s //tensorflow/dtensor/python/tests:mesh_util_test_cpu PASSED in 14.2s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 17.4s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 13.4s //tensorflow/dtensor/python/tests:variable_test_cpu PASSED in 12.4s //tensorflow/dtensor/tests:dtensor_operation_test PASSED in 26.8s //tensorflow/dtensor/tests:executable_manager_test PASSED in 38.0s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.2s //tensorflow/dtensor/tests:slice_util_test PASSED in 0.6s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 9.4s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.3s //tensorflow/examples/adding_an_op:fact_test PASSED in 42.6s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 24.0s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 37.6s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 25.3s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 38.1s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 22.5s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 22.9s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 24.3s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 27.5s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 27.9s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 2.3s //tensorflow/examples/speech_commands:models_test PASSED in 30.2s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 1.4s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.4s //tensorflow/js:ts_op_gen_test PASSED in 0.4s //tensorflow/python/autograph/converters:asserts_test PASSED in 23.0s //tensorflow/python/autograph/converters:break_statements_test PASSED in 9.3s //tensorflow/python/autograph/converters:call_trees_test PASSED in 9.7s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 9.4s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 12.0s //tensorflow/python/autograph/converters:control_flow_test PASSED in 17.2s //tensorflow/python/autograph/converters:directives_test PASSED in 21.6s //tensorflow/python/autograph/converters:functions_test PASSED in 9.0s //tensorflow/python/autograph/converters:lists_test PASSED in 9.4s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 9.4s //tensorflow/python/autograph/converters:return_statements_test PASSED in 12.4s //tensorflow/python/autograph/converters:slices_test PASSED in 10.8s //tensorflow/python/autograph/converters:variables_test PASSED in 8.2s //tensorflow/python/autograph/core:converter_test PASSED in 8.7s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 27.9s //tensorflow/python/autograph/impl:api_test PASSED in 23.2s //tensorflow/python/autograph/impl:conversion_test PASSED in 10.4s //tensorflow/python/autograph/lang:special_functions_test PASSED in 17.2s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 10.3s //tensorflow/python/autograph/operators:control_flow_test PASSED in 19.3s //tensorflow/python/autograph/operators:data_structures_test PASSED in 9.4s //tensorflow/python/autograph/operators:exceptions_test PASSED in 10.0s //tensorflow/python/autograph/operators:logical_test PASSED in 9.3s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 23.6s //tensorflow/python/autograph/operators:slices_test PASSED in 10.3s //tensorflow/python/autograph/operators:variables_test PASSED in 10.5s //tensorflow/python/autograph/pyct:anno_test PASSED in 11.1s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 10.8s //tensorflow/python/autograph/pyct:cache_test PASSED in 8.6s //tensorflow/python/autograph/pyct:cfg_test PASSED in 9.5s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 9.9s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 10.6s //tensorflow/python/autograph/pyct:loader_test PASSED in 11.4s //tensorflow/python/autograph/pyct:naming_test PASSED in 11.6s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 12.6s //tensorflow/python/autograph/pyct:parser_test PASSED in 10.3s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 10.2s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 9.7s //tensorflow/python/autograph/pyct:templates_test PASSED in 11.2s //tensorflow/python/autograph/pyct:transformer_test PASSED in 12.5s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 12.0s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 9.7s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 8.9s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 10.1s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 12.0s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 12.0s //tensorflow/python/autograph/tests:assertion_test PASSED in 21.1s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 25.6s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 41.6s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 25.3s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 22.1s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 22.1s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 28.9s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 43.3s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 41.0s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 30.3s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 35.2s //tensorflow/python/autograph/tests:datasets_test PASSED in 52.3s //tensorflow/python/autograph/tests:early_return_test PASSED in 42.6s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 21.7s //tensorflow/python/autograph/tests:generator_test PASSED in 22.3s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 30.4s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 84.9s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 24.7s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 47.1s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 30.1s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 32.7s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 29.1s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 48.4s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 57.4s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 23.2s //tensorflow/python/autograph/utils:context_managers_test PASSED in 9.2s //tensorflow/python/autograph/utils:misc_test PASSED in 10.4s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 23.9s //tensorflow/python/autograph/utils:tensors_test PASSED in 10.4s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 20.3s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 18.6s //tensorflow/python/checkpoint:checkpoint_test PASSED in 33.2s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 10.8s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 15.2s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 13.2s //tensorflow/python/checkpoint:restore_test PASSED in 9.5s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 9.8s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 10.9s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 9.9s //tensorflow/python/checkpoint:trackable_view_test PASSED in 9.1s //tensorflow/python/checkpoint/sharding:sharding_policies_test PASSED in 12.7s //tensorflow/python/checkpoint/sharding:sharding_util_test PASSED in 12.1s //tensorflow/python/client:device_lib_test_cpu PASSED in 11.0s //tensorflow/python/client:events_writer_test PASSED in 11.4s //tensorflow/python/client:session_list_devices_test PASSED in 10.6s //tensorflow/python/client:session_partial_run_test PASSED in 12.9s //tensorflow/python/client:timeline_test_cpu PASSED in 9.6s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 9.0s //tensorflow/python/compat:compat_test PASSED in 10.4s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 8.5s //tensorflow/python/compiler/mlir:mlir_test PASSED in 8.8s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 29.3s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 11.8s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 11.5s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 10.6s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 18.1s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 12.3s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 29.3s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 14.3s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 38.1s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 12.0s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 11.9s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 11.0s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 13.8s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 31.7s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 12.5s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 15.6s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 15.5s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 13.8s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 11.3s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 11.5s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 10.3s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 11.6s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 10.5s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 19.0s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 27.6s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 27.1s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 23.7s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 13.9s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 13.2s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 23.9s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 16.0s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 17.4s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 21.7s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 31.3s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 32.0s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 12.9s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 21.9s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 10.2s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 57.0s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 10.0s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 18.2s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 11.9s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 14.1s //tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 30.3s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 15.3s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 14.1s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 25.9s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 10.7s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 14.9s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 10.7s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 11.8s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 39.4s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 46.3s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 13.7s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 33.0s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 11.2s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 23.5s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 154.5s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 29.6s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 15.3s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 17.0s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 10.7s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 12.6s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 22.8s //tensorflow/python/data/kernel_tests:cache_test PASSED in 48.6s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 16.1s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 25.2s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 33.5s //tensorflow/python/data/kernel_tests:counter_test PASSED in 41.3s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 12.2s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 58.9s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 24.2s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 10.6s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 58.6s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 23.8s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 15.2s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 29.5s //tensorflow/python/data/kernel_tests:io_test PASSED in 51.1s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 22.1s //tensorflow/python/data/kernel_tests:len_test PASSED in 13.5s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 14.3s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 13.7s //tensorflow/python/data/kernel_tests:options_test PASSED in 17.3s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 12.5s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 64.3s //tensorflow/python/data/kernel_tests:random_test PASSED in 30.5s //tensorflow/python/data/kernel_tests:range_test PASSED in 43.6s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 26.9s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 31.2s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 62.1s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 34.0s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 23.7s //tensorflow/python/data/util:convert_test PASSED in 10.7s //tensorflow/python/data/util:nest_test PASSED in 11.0s //tensorflow/python/data/util:options_test PASSED in 13.7s //tensorflow/python/data/util:random_seed_test PASSED in 12.9s //tensorflow/python/data/util:sparse_test PASSED in 11.4s //tensorflow/python/data/util:structure_test PASSED in 12.3s //tensorflow/python/data/util:traverse_test PASSED in 12.1s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 12.0s //tensorflow/python/debug/cli:cli_config_test PASSED in 9.1s //tensorflow/python/debug/cli:cli_shared_test PASSED in 10.7s //tensorflow/python/debug/cli:command_parser_test PASSED in 9.1s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 9.3s //tensorflow/python/debug/cli:evaluator_test PASSED in 10.3s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 10.1s //tensorflow/python/debug/cli:readline_ui_test PASSED in 10.3s //tensorflow/python/debug/cli:tensor_format_test PASSED in 9.3s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 29.7s //tensorflow/python/debug/lib:common_test PASSED in 8.7s //tensorflow/python/debug/lib:debug_data_test PASSED in 9.4s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 11.6s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 15.3s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 12.4s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 12.5s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 10.6s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 14.8s //tensorflow/python/debug/lib:debug_utils_test PASSED in 9.7s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 22.2s //tensorflow/python/debug/lib:profiling_test PASSED in 7.7s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 46.3s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 14.4s //tensorflow/python/debug/lib:source_utils_test PASSED in 12.8s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 10.7s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 12.0s //tensorflow/python/debug/wrappers:framework_test PASSED in 12.2s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 10.3s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 36.2s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 16.1s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 13.0s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 14.4s //tensorflow/python/distribute:collective_util_test PASSED in 9.7s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 29.8s //tensorflow/python/distribute:combinations_test_cpu PASSED in 27.8s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 13.3s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 15.2s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 11.7s //tensorflow/python/distribute:device_util_test_cpu PASSED in 17.7s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 16.6s //tensorflow/python/distribute:distribute_lib_test PASSED in 17.5s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 13.2s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 18.5s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 16.3s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 32.8s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 35.7s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 16.4s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 18.9s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 33.4s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 23.9s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 13.5s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 28.6s //tensorflow/python/distribute:multi_worker_util_test PASSED in 10.3s //tensorflow/python/distribute:numpy_dataset_test PASSED in 10.1s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 23.3s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 10.4s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 36.7s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 58.7s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 30.2s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 28.5s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 21.1s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 16.2s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 14.9s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 18.2s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 13.3s //tensorflow/python/distribute:sharded_variable_test PASSED in 21.3s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 29.4s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 52.0s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 16.3s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 22.9s //tensorflow/python/distribute:test_util_test_cpu PASSED in 25.2s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 12.9s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 14.9s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 16.3s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 14.0s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 14.1s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 10.3s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 10.0s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 11.6s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 9.8s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 10.9s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 9.4s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 11.6s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 63.9s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 14.4s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 42.2s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 20.8s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 87.3s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 17.8s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 48.6s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 96.8s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 12.0s //tensorflow/python/eager:backprop_test_cpu PASSED in 153.0s //tensorflow/python/eager:cancellation_test_cpu PASSED in 9.1s //tensorflow/python/eager:context_test_cpu PASSED in 18.9s //tensorflow/python/eager:core_test_cpu PASSED in 22.0s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 60.6s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 10.2s //tensorflow/python/eager:lift_to_graph_test PASSED in 10.9s //tensorflow/python/eager:monitoring_test_cpu PASSED in 12.3s //tensorflow/python/eager:ops_test_cpu PASSED in 9.3s //tensorflow/python/eager:profiler_client_test PASSED in 9.1s //tensorflow/python/eager:profiler_test_cpu PASSED in 10.1s //tensorflow/python/eager:pywrap_tfe_test PASSED in 31.3s //tensorflow/python/eager:record_test PASSED in 16.7s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 12.2s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 8.7s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 225.7s //tensorflow/python/eager:tensor_test_cpu PASSED in 14.6s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 12.5s //tensorflow/python/eager:wrap_function_test PASSED in 17.0s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 9.1s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 9.5s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 32.1s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 9.4s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 19.9s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 14.0s //tensorflow/python/eager/polymorphic_function:concrete_function_test_cpu PASSED in 10.6s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 9.0s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 29.4s //tensorflow/python/eager/polymorphic_function:tracing_compilation_test PASSED in 18.5s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 14.0s //tensorflow/python/feature_column:serialization_test PASSED in 17.9s //tensorflow/python/framework:auto_control_deps_test PASSED in 35.2s //tensorflow/python/framework:c_api_util_test PASSED in 8.9s //tensorflow/python/framework:common_shapes_test PASSED in 34.7s //tensorflow/python/framework:composite_tensor_test PASSED in 10.8s //tensorflow/python/framework:config_test_2gpu PASSED in 14.6s //tensorflow/python/framework:config_test_cpu PASSED in 14.7s //tensorflow/python/framework:constant_op_test PASSED in 20.0s //tensorflow/python/framework:device_spec_test PASSED in 10.2s //tensorflow/python/framework:device_test PASSED in 9.0s //tensorflow/python/framework:dtypes_test PASSED in 31.6s //tensorflow/python/framework:error_interpolation_test PASSED in 12.1s //tensorflow/python/framework:errors_test PASSED in 13.2s //tensorflow/python/framework:extension_type_field_test PASSED in 9.3s //tensorflow/python/framework:extension_type_test PASSED in 35.3s //tensorflow/python/framework:file_system_test PASSED in 11.3s //tensorflow/python/framework:flexible_dtypes_test PASSED in 153.1s //tensorflow/python/framework:function_def_to_graph_test PASSED in 31.7s //tensorflow/python/framework:graph_util_test PASSED in 12.3s //tensorflow/python/framework:immutable_dict_test PASSED in 10.2s //tensorflow/python/framework:importer_test PASSED in 12.3s //tensorflow/python/framework:indexed_slices_test PASSED in 10.8s //tensorflow/python/framework:kernels_test PASSED in 9.1s //tensorflow/python/framework:meta_graph_test PASSED in 18.6s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 9.5s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.3s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 3.8s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 15.3s //tensorflow/python/framework:op_def_library_test PASSED in 11.7s //tensorflow/python/framework:op_def_util_test PASSED in 11.7s //tensorflow/python/framework:ops_enable_eager_test PASSED in 3.5s //tensorflow/python/framework:ops_test PASSED in 27.3s //tensorflow/python/framework:proto_test PASSED in 27.6s //tensorflow/python/framework:py_context_manager_test PASSED in 9.8s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 11.9s //tensorflow/python/framework:python_api_info_test PASSED in 9.0s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 13.1s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 5.3s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.1s //tensorflow/python/framework:python_op_gen_test PASSED in 0.1s //tensorflow/python/framework:python_tensor_converter_test PASSED in 10.0s //tensorflow/python/framework:random_seed_test PASSED in 12.2s //tensorflow/python/framework:registry_test PASSED in 9.5s //tensorflow/python/framework:smart_cond_test PASSED in 10.9s //tensorflow/python/framework:sparse_tensor_test PASSED in 13.5s //tensorflow/python/framework:subscribe_test PASSED in 13.5s //tensorflow/python/framework:tensor_shape_test PASSED in 9.5s //tensorflow/python/framework:tensor_test PASSED in 13.2s //tensorflow/python/framework:tensor_util_test PASSED in 13.7s //tensorflow/python/framework:test_combinations_test PASSED in 10.5s //tensorflow/python/framework:test_util_test_cpu PASSED in 18.0s //tensorflow/python/framework:tf2_test PASSED in 10.6s //tensorflow/python/framework:traceable_stack_test PASSED in 12.8s //tensorflow/python/framework:type_spec_test PASSED in 10.1s //tensorflow/python/framework:versions_test PASSED in 9.0s //tensorflow/python/framework:weak_tensor_test PASSED in 14.8s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 13.1s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 9.6s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 15.2s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 22.5s //tensorflow/python/grappler:cost_analyzer_test PASSED in 13.7s //tensorflow/python/grappler:datasets_test PASSED in 24.4s //tensorflow/python/grappler:item_test PASSED in 10.2s //tensorflow/python/grappler:memory_optimizer_test PASSED in 20.1s //tensorflow/python/grappler:model_analyzer_test PASSED in 9.8s //tensorflow/python/grappler:remapper_test_cpu PASSED in 10.6s //tensorflow/python/grappler:tf_optimizer_test PASSED in 10.2s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 19.5s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 35.1s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 12.1s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 23.8s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 10.5s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 20.0s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests:template_test PASSED in 15.4s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 13.4s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 17.9s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 9.4s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 30.0s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 31.0s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 11.0s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 9.5s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 10.0s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 10.7s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 29.6s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 17.8s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 16.6s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 18.6s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 19.5s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 35.0s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 18.0s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 23.3s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 66.5s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 20.5s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 10.0s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 23.5s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 95.8s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 80.0s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 12.6s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 20.6s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 11.6s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 16.6s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 15.3s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 19.4s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 15.5s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 11.2s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 46.1s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 12.0s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 26.9s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 15.5s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 10.1s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 29.0s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 16.2s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 22.8s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 40.1s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 21.7s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 52.6s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 30.5s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 35.9s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 25.0s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 37.0s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 31.6s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 9.7s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 31.7s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 9.5s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 12.5s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 11.0s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 11.3s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 8.8s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 12.5s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 11.2s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 33.4s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 11.9s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 53.3s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 10.0s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 59.9s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 21.1s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 17.4s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 14.7s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 8.9s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 13.1s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 11.1s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 13.3s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 8.6s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 28.9s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 37.0s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 13.1s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 137.6s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 12.0s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 98.8s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 57.6s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 14.6s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 15.0s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 35.8s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 23.7s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 23.2s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 29.5s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 125.5s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/proto:decode_proto_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 20.0s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 10.2s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 19.9s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 14.3s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 11.0s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 18.6s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 20.7s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 18.0s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 10.9s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 21.9s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 155.3s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 24.1s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 28.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 11.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 11.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 17.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 38.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 10.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 10.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 13.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 37.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 43.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 25.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 10.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 73.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 13.9s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 11.4s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 15.1s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 12.1s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 10.2s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 10.7s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 10.0s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 13.3s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 10.8s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 17.7s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 12.2s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 10.0s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 22.7s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 31.4s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 15.5s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 12.7s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 12.8s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 10.1s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 8.6s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 12.8s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 13.6s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 57.2s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 44.4s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 16.0s //tensorflow/python/lib/io:file_io_test PASSED in 10.7s //tensorflow/python/lib/io:tf_record_test PASSED in 11.2s //tensorflow/python/module:module_test PASSED in 28.2s //tensorflow/python/ops:array_grad_test_cpu PASSED in 12.6s //tensorflow/python/ops:array_ops_shape_test PASSED in 12.6s //tensorflow/python/ops:array_ops_test PASSED in 9.3s //tensorflow/python/ops:autograph_ops_test PASSED in 10.3s //tensorflow/python/ops:bincount_ops_test_cpu PASSED in 14.7s //tensorflow/python/ops:bitwise_ops_test_cpu PASSED in 11.2s //tensorflow/python/ops:clip_ops_test PASSED in 15.4s //tensorflow/python/ops:clustering_ops_test PASSED in 23.6s //tensorflow/python/ops:collective_ops_gpu_test_cpu PASSED in 11.6s //tensorflow/python/ops:collective_ops_test PASSED in 32.4s //tensorflow/python/ops:collective_ops_xla_test PASSED in 11.0s //tensorflow/python/ops:compiled_collective_ops_gpu_test_2gpu PASSED in 12.3s //tensorflow/python/ops:compiled_collective_ops_gpu_test_cpu PASSED in 15.1s //tensorflow/python/ops:control_flow_v2_enable_test PASSED in 10.0s //tensorflow/python/ops:control_flow_v2_toggles_test PASSED in 10.5s //tensorflow/python/ops:dequantize_op_test PASSED in 14.5s //tensorflow/python/ops:embedding_ops_test_cpu PASSED in 11.5s //tensorflow/python/ops:factory_ops_test_cpu PASSED in 11.8s //tensorflow/python/ops:functional_ops_test PASSED in 14.7s //tensorflow/python/ops:gradient_checker_v2_test_cpu PASSED in 30.8s //tensorflow/python/ops:gradients_test_cpu PASSED in 19.7s //tensorflow/python/ops:init_ops_test_cpu PASSED in 11.6s //tensorflow/python/ops:init_ops_v2_test_cpu PASSED in 11.7s //tensorflow/python/ops:lookup_ops_async_checkpoint_test PASSED in 12.7s //tensorflow/python/ops:math_grad_test_cpu PASSED in 20.5s //tensorflow/python/ops:math_ops_linspace_test_cpu PASSED in 12.4s //tensorflow/python/ops:math_ops_test_cpu PASSED in 28.6s //tensorflow/python/ops:nn_grad_test_cpu PASSED in 17.5s //tensorflow/python/ops:nn_loss_scaling_utilities_test PASSED in 12.5s //tensorflow/python/ops:nn_test_cpu PASSED in 87.8s //tensorflow/python/ops:nn_xent_test_cpu PASSED in 13.1s //tensorflow/python/ops:op_selector_test PASSED in 11.3s //tensorflow/python/ops:quantized_conv_ops_test PASSED in 8.9s //tensorflow/python/ops:quantized_ops_test PASSED in 10.4s //tensorflow/python/ops:raw_ops_test_cpu PASSED in 11.9s //tensorflow/python/ops:rnn_grad_test_cpu PASSED in 9.5s //tensorflow/python/ops:script_ops_test PASSED in 10.7s //tensorflow/python/ops:sort_ops_test PASSED in 10.7s //tensorflow/python/ops:sparse_bincount_ops_test_cpu PASSED in 15.3s //tensorflow/python/ops:sparse_ops_test PASSED in 21.7s //tensorflow/python/ops:tensor_array_ops_test PASSED in 10.9s //tensorflow/python/ops:variable_spec_test PASSED in 10.8s //tensorflow/python/ops:weak_tensor_array_ops_test PASSED in 9.5s //tensorflow/python/ops:weak_tensor_constant_op_test PASSED in 15.2s //tensorflow/python/ops:weak_tensor_image_ops_test PASSED in 10.6s //tensorflow/python/ops:weak_tensor_math_ops_test PASSED in 42.2s //tensorflow/python/ops:weak_tensor_nn_test_cpu PASSED in 17.9s //tensorflow/python/ops:weak_tensor_np_array_ops_test PASSED in 45.2s //tensorflow/python/ops:weak_tensor_np_math_ops_test PASSED in 12.1s //tensorflow/python/ops:weak_tensor_ops_test PASSED in 129.8s //tensorflow/python/ops/losses:util_test PASSED in 13.2s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 15.4s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 109.7s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 11.4s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 9.9s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 46.9s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 11.9s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 28.8s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 64.1s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 9.0s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 22.9s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 22.2s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 45.3s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 18.0s //tensorflow/python/ops/parallel_for:pfor_test PASSED in 9.0s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 47.4s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 14.5s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 50.7s //tensorflow/python/ops/ragged:ragged_bincount_ops_test_cpu PASSED in 13.4s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 12.4s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 16.9s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 16.5s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 12.5s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 9.9s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 27.7s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 102.5s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 39.3s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 9.3s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 12.1s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 17.2s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 12.0s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 11.0s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 28.3s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 10.3s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 13.6s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 21.4s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 21.2s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 38.4s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 28.7s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 12.2s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 24.6s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 9.2s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 20.3s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 9.3s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 12.3s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 55.9s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 22.0s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 12.2s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 14.4s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 11.1s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 10.8s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 40.8s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 10.0s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 50.7s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 18.9s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 15.9s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 12.6s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 63.4s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 52.5s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 11.9s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 81.7s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 24.5s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 33.8s //tensorflow/python/ops/ragged:row_partition_test PASSED in 47.8s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 9.3s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 11.2s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 55.9s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 62.7s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 16.3s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 66.9s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 14.6s //tensorflow/python/platform:benchmark_test PASSED in 11.3s //tensorflow/python/platform:build_info_test PASSED in 9.8s //tensorflow/python/platform:resource_loader_test PASSED in 3.5s //tensorflow/python/profiler:pprof_profiler_test PASSED in 10.5s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 27.3s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 12.4s //tensorflow/python/profiler:profiler_test_cpu PASSED in 20.3s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 9.6s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 10.3s //tensorflow/python/profiler:tfprof_logger_test PASSED in 12.3s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 10.4s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 10.2s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 18.4s //tensorflow/python/saved_model:fingerprinting_test PASSED in 11.0s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 19.5s //tensorflow/python/saved_model:loader_test PASSED in 16.5s //tensorflow/python/saved_model:method_name_updater_test PASSED in 9.1s //tensorflow/python/saved_model:metrics_test PASSED in 11.9s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 13.4s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 9.1s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 10.2s //tensorflow/python/saved_model:revived_types_test PASSED in 13.4s //tensorflow/python/saved_model:save_context_test PASSED in 11.2s //tensorflow/python/saved_model:save_test PASSED in 37.8s //tensorflow/python/saved_model:saved_model_test PASSED in 26.2s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 24.6s //tensorflow/python/saved_model:simple_save_test PASSED in 28.6s //tensorflow/python/saved_model:tracing_utils_test PASSED in 11.0s //tensorflow/python/saved_model:utils_test PASSED in 25.6s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 10.5s //tensorflow/python/saved_model/model_utils:export_test PASSED in 13.6s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 13.3s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 23.2s //tensorflow/python/saved_model/registration:registration_test PASSED in 9.4s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 38.2s //tensorflow/python/saved_model/tests:variable_wrapper_test PASSED in 13.3s //tensorflow/python/summary:plugin_asset_test PASSED in 9.6s //tensorflow/python/summary:summary_iterator_test PASSED in 22.8s //tensorflow/python/summary:summary_test PASSED in 11.9s //tensorflow/python/summary:summary_v2_test PASSED in 27.5s //tensorflow/python/summary/writer:writer_test PASSED in 20.3s //tensorflow/python/tools:aot_compiled_test PASSED in 21.3s //tensorflow/python/tools:freeze_graph_test PASSED in 12.1s //tensorflow/python/tools:optimize_for_inference_test PASSED in 9.9s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 36.4s //tensorflow/python/tools:saved_model_cli_test PASSED in 34.9s //tensorflow/python/tools:saved_model_utils_test PASSED in 24.5s //tensorflow/python/tools:strip_unused_test PASSED in 8.9s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 26.6s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 18.6s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 9.8s //tensorflow/python/tools/api/generator2/extractor:extractor_test PASSED in 1.2s //tensorflow/python/tools/api/generator2/generator:generator_test PASSED in 1.6s //tensorflow/python/tools/api/generator2/shared:exported_api_test PASSED in 25.7s //tensorflow/python/tpu:bfloat16_test PASSED in 9.6s //tensorflow/python/tpu:feature_column_test PASSED in 26.0s //tensorflow/python/tpu:topology_test PASSED in 16.4s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 12.7s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 11.7s //tensorflow/python/tpu:tpu_embedding_v3_utils_test PASSED in 9.7s //tensorflow/python/tpu:tpu_infeed_test PASSED in 15.1s //tensorflow/python/tpu:tpu_sharding_test PASSED in 15.6s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 9.4s //tensorflow/python/tpu/client:client_py_test PASSED in 9.6s //tensorflow/python/trackable:autotrackable_test PASSED in 11.6s //tensorflow/python/trackable:base_delegate_test PASSED in 14.5s //tensorflow/python/trackable:base_test PASSED in 9.9s //tensorflow/python/trackable:python_state_test PASSED in 8.5s //tensorflow/python/trackable:resource_test PASSED in 11.4s //tensorflow/python/trackable:trackable_utils_test PASSED in 8.9s //tensorflow/python/training:adadelta_test_cpu PASSED in 19.3s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 12.7s //tensorflow/python/training:adagrad_test_cpu PASSED in 17.3s //tensorflow/python/training:adam_test_cpu PASSED in 18.0s //tensorflow/python/training:basic_loops_test_cpu PASSED in 12.2s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 29.4s //tensorflow/python/training:checkpoint_ops_test PASSED in 10.6s //tensorflow/python/training:coordinator_test_cpu PASSED in 18.8s //tensorflow/python/training:device_setter_test_cpu PASSED in 10.6s //tensorflow/python/training:ftrl_test_cpu PASSED in 22.0s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 22.5s //tensorflow/python/training:input_test PASSED in 30.3s //tensorflow/python/training:momentum_test_cpu PASSED in 16.0s //tensorflow/python/training:monitored_session_test PASSED in 30.9s //tensorflow/python/training:moving_averages_test_cpu PASSED in 20.4s //tensorflow/python/training:optimizer_test_cpu PASSED in 15.7s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 13.9s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 11.1s //tensorflow/python/training:quantize_training_test_cpu PASSED in 9.6s //tensorflow/python/training:queue_runner_test_cpu PASSED in 12.7s //tensorflow/python/training:rmsprop_test_cpu PASSED in 41.9s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 17.4s //tensorflow/python/training:saver_test_2gpu PASSED in 34.8s //tensorflow/python/training:saver_test_cpu PASSED in 38.7s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 10.2s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 11.0s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 10.6s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 9.3s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 10.6s //tensorflow/python/training:server_lib_test PASSED in 20.3s //tensorflow/python/training:session_manager_test_cpu PASSED in 84.0s //tensorflow/python/training:slot_creator_test_cpu PASSED in 16.1s //tensorflow/python/training:supervisor_test PASSED in 28.0s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 29.5s //tensorflow/python/training:training_ops_test_cpu PASSED in 10.1s //tensorflow/python/training:training_util_test PASSED in 12.7s //tensorflow/python/training:warm_starting_util_test PASSED in 27.0s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 17.1s //tensorflow/python/training/experimental:loss_scale_test PASSED in 25.3s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 11.5s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 9.4s //tensorflow/python/util:compat_test PASSED in 13.2s //tensorflow/python/util:decorator_utils_test PASSED in 9.8s //tensorflow/python/util:deprecation_test PASSED in 10.8s //tensorflow/python/util:dispatch_test PASSED in 30.6s //tensorflow/python/util:example_parser_configuration_test PASSED in 19.2s //tensorflow/python/util:fast_module_type_test PASSED in 9.3s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 10.1s //tensorflow/python/util:function_utils_test PASSED in 9.7s //tensorflow/python/util:keyword_args_test PASSED in 10.0s //tensorflow/python/util:lazy_loader_test PASSED in 11.2s //tensorflow/python/util:lock_util_test PASSED in 17.7s //tensorflow/python/util:module_wrapper_test PASSED in 10.5s //tensorflow/python/util:nest_test PASSED in 29.9s //tensorflow/python/util:object_identity_test PASSED in 9.6s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 4.3s //tensorflow/python/util:serialization_test PASSED in 28.0s //tensorflow/python/util:tf_contextlib_test PASSED in 9.5s //tensorflow/python/util:tf_decorator_test PASSED in 9.6s //tensorflow/python/util:tf_export_test PASSED in 12.4s //tensorflow/python/util:tf_inspect_test PASSED in 13.9s //tensorflow/python/util:tf_should_use_test PASSED in 13.4s //tensorflow/python/util:tf_stack_test PASSED in 11.5s //tensorflow/python/util:traceback_utils_test PASSED in 8.3s //tensorflow/python/util:type_annotations_test PASSED in 10.6s //tensorflow/python/util:variable_utils_test PASSED in 13.6s //tensorflow/python/util:vlog_test PASSED in 9.5s //tensorflow/python/util/protobuf:protobuf_compare_test PASSED in 6.1s //tensorflow/tools/api/tests:module_test PASSED in 18.5s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.3s //tensorflow/tools/common:public_api_test PASSED in 2.9s //tensorflow/tools/common:traverse_test PASSED in 4.5s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 11.7s //tensorflow/tools/compatibility:ast_edits_test PASSED in 15.5s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 21.9s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 25.3s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 11.2s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 8.7s //tensorflow/tools/docs:tf_doctest_test PASSED in 1.7s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.7s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 2.0s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 2.2s //tensorflow/tools/graph_transforms:transforms_test PASSED in 3.2s //tensorflow/tools/proto_splitter:merge_test PASSED in 0.2s //tensorflow/tools/proto_splitter:split_graph_def_test PASSED in 9.3s //tensorflow/tools/proto_splitter:split_test PASSED in 9.8s //tensorflow/tools/proto_splitter:util_test PASSED in 9.5s //tensorflow/tools/proto_splitter/cc:composable_splitter_test PASSED in 0.6s //tensorflow/tools/proto_splitter/cc:graph_def_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:saved_model_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:util_test PASSED in 2.9s //tensorflow/tools/proto_splitter/python:saved_model_test PASSED in 14.0s //tensorflow/tools/proto_splitter/python:test_util_test PASSED in 10.1s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.2s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 1.6s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 12.2s Stats over 2 runs: max = 12.2s, min = 11.3s, avg = 11.7s, dev = 0.5s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 11.6s Stats over 2 runs: max = 11.6s, min = 9.9s, avg = 10.7s, dev = 0.9s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 20.8s Stats over 2 runs: max = 20.8s, min = 14.7s, avg = 17.8s, dev = 3.0s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 21.0s Stats over 2 runs: max = 21.0s, min = 19.0s, avg = 20.0s, dev = 1.0s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 36.1s Stats over 2 runs: max = 36.1s, min = 33.3s, avg = 34.7s, dev = 1.4s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 51.6s Stats over 2 runs: max = 51.6s, min = 47.3s, avg = 49.4s, dev = 2.1s //tensorflow/python/data/kernel_tests:window_test PASSED in 37.6s Stats over 2 runs: max = 37.6s, min = 30.5s, avg = 34.0s, dev = 3.6s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 18.2s Stats over 2 runs: max = 18.2s, min = 16.3s, avg = 17.2s, dev = 0.9s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 21.5s Stats over 2 runs: max = 21.5s, min = 21.3s, avg = 21.4s, dev = 0.1s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 16.4s Stats over 2 runs: max = 16.4s, min = 15.5s, avg = 16.0s, dev = 0.5s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 57.7s Stats over 2 runs: max = 57.7s, min = 29.3s, avg = 43.5s, dev = 14.2s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 112.8s Stats over 2 runs: max = 112.8s, min = 46.7s, avg = 79.7s, dev = 33.0s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 10.0s Stats over 2 runs: max = 10.0s, min = 9.6s, avg = 9.8s, dev = 0.2s //tensorflow/python/ops:control_flow_ops_test_cpu PASSED in 31.9s Stats over 2 runs: max = 31.9s, min = 27.2s, avg = 29.5s, dev = 2.3s //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 32.1s Stats over 3 runs: max = 32.1s, min = 30.4s, avg = 31.3s, dev = 0.7s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 19.9s Stats over 3 runs: max = 19.9s, min = 19.0s, avg = 19.5s, dev = 0.4s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.3s, dev = 0.0s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 23.8s Stats over 3 runs: max = 23.8s, min = 15.9s, avg = 21.1s, dev = 3.7s //tensorflow/python/data/kernel_tests:unique_test PASSED in 36.2s Stats over 3 runs: max = 36.2s, min = 32.9s, avg = 34.2s, dev = 1.4s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 23.8s Stats over 3 runs: max = 23.8s, min = 19.2s, avg = 21.7s, dev = 1.9s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 54.9s Stats over 3 runs: max = 54.9s, min = 32.3s, avg = 40.1s, dev = 10.4s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 11.0s Stats over 3 runs: max = 11.0s, min = 9.9s, avg = 10.5s, dev = 0.5s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 18.9s Stats over 3 runs: max = 18.9s, min = 17.8s, avg = 18.3s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 146.0s Stats over 3 runs: max = 146.0s, min = 14.6s, avg = 59.4s, dev = 61.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 10.1s Stats over 3 runs: max = 10.1s, min = 8.7s, avg = 9.4s, dev = 0.6s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 18.0s Stats over 3 runs: max = 18.0s, min = 13.9s, avg = 15.5s, dev = 1.8s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test FAILED in 3 out of 3 in 22.7s Stats over 3 runs: max = 22.7s, min = 18.0s, avg = 20.7s, dev = 2.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/mlir/quantization/stablehlo/convert_tf_quant_to_mhlo_int_test/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/mlir/quantization/stablehlo/convert_tf_quant_to_mhlo_int_test/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/mlir/quantization/stablehlo/convert_tf_quant_to_mhlo_int_test/test_attempts/attempt_2.log //tensorflow/core/kernels:example_parsing_ops_test PASSED in 0.6s Stats over 4 runs: max = 0.6s, min = 0.5s, avg = 0.6s, dev = 0.1s //tensorflow/dtensor/python/tests:batchparallel_spmd_test_cpu PASSED in 19.0s Stats over 4 runs: max = 19.0s, min = 17.2s, avg = 18.3s, dev = 0.7s //tensorflow/dtensor/python/tests:conv_test_cpu PASSED in 14.3s Stats over 4 runs: max = 14.3s, min = 13.5s, avg = 13.8s, dev = 0.3s //tensorflow/dtensor/python/tests:sparse_test_cpu PASSED in 28.0s Stats over 4 runs: max = 28.0s, min = 13.0s, avg = 22.2s, dev = 5.6s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 34.9s Stats over 4 runs: max = 34.9s, min = 19.0s, avg = 28.1s, dev = 6.2s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 64.9s Stats over 4 runs: max = 64.9s, min = 45.0s, avg = 51.1s, dev = 8.0s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 31.4s Stats over 4 runs: max = 31.4s, min = 16.7s, avg = 24.2s, dev = 6.8s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 36.8s Stats over 4 runs: max = 36.8s, min = 9.3s, avg = 21.7s, dev = 12.0s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 46.8s Stats over 4 runs: max = 46.8s, min = 27.3s, avg = 34.5s, dev = 7.4s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 11.3s Stats over 4 runs: max = 11.3s, min = 10.3s, avg = 10.8s, dev = 0.4s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 80.0s Stats over 4 runs: max = 80.0s, min = 46.2s, avg = 62.6s, dev = 15.6s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 18.9s Stats over 4 runs: max = 18.9s, min = 11.4s, avg = 14.8s, dev = 3.4s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 44.5s Stats over 4 runs: max = 44.5s, min = 24.7s, avg = 33.9s, dev = 7.1s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 26.2s Stats over 4 runs: max = 26.2s, min = 11.0s, avg = 17.3s, dev = 6.6s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 25.1s Stats over 4 runs: max = 25.1s, min = 21.7s, avg = 23.4s, dev = 1.3s //tensorflow/python/data/kernel_tests:skip_test PASSED in 33.1s Stats over 4 runs: max = 33.1s, min = 22.1s, avg = 28.0s, dev = 4.5s //tensorflow/python/data/kernel_tests:take_test PASSED in 30.3s Stats over 4 runs: max = 30.3s, min = 25.3s, avg = 27.3s, dev = 1.8s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 33.4s Stats over 4 runs: max = 33.4s, min = 30.8s, avg = 31.8s, dev = 1.0s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 25.1s Stats over 4 runs: max = 25.1s, min = 18.9s, avg = 22.1s, dev = 2.9s //tensorflow/python/data/kernel_tests:zip_test PASSED in 37.0s Stats over 4 runs: max = 37.0s, min = 33.7s, avg = 35.0s, dev = 1.3s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 17.4s Stats over 4 runs: max = 17.4s, min = 16.7s, avg = 17.0s, dev = 0.3s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 46.7s Stats over 4 runs: max = 46.7s, min = 37.0s, avg = 41.6s, dev = 3.7s //tensorflow/python/framework:convert_to_constants_test PASSED in 48.1s Stats over 4 runs: max = 48.1s, min = 40.3s, avg = 43.8s, dev = 2.9s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 42.6s Stats over 4 runs: max = 42.6s, min = 40.0s, avg = 41.6s, dev = 1.0s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 19.6s Stats over 4 runs: max = 19.6s, min = 11.1s, avg = 14.5s, dev = 3.1s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 63.2s Stats over 4 runs: max = 63.2s, min = 30.3s, avg = 45.2s, dev = 13.5s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 36.8s Stats over 4 runs: max = 36.8s, min = 13.5s, avg = 23.7s, dev = 9.9s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 76.6s Stats over 4 runs: max = 76.6s, min = 14.0s, avg = 37.9s, dev = 25.2s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 52.9s Stats over 4 runs: max = 52.9s, min = 50.7s, avg = 51.6s, dev = 0.8s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 47.4s Stats over 4 runs: max = 47.4s, min = 36.9s, avg = 41.0s, dev = 4.2s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 135.0s Stats over 4 runs: max = 135.0s, min = 15.2s, avg = 71.2s, dev = 54.5s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 18.2s Stats over 4 runs: max = 18.2s, min = 17.6s, avg = 17.9s, dev = 0.2s //tensorflow/python/ops:nn_batchnorm_test_cpu PASSED in 20.6s Stats over 4 runs: max = 20.6s, min = 16.0s, avg = 17.6s, dev = 1.8s //tensorflow/python/ops:nn_fused_batchnorm_d9m_test_cpu PASSED in 32.2s Stats over 4 runs: max = 32.2s, min = 30.9s, avg = 31.5s, dev = 0.7s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 73.2s Stats over 4 runs: max = 73.2s, min = 26.8s, avg = 53.6s, dev = 17.1s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 66.3s Stats over 4 runs: max = 66.3s, min = 58.2s, avg = 62.3s, dev = 3.3s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 18.5s Stats over 5 runs: max = 18.5s, min = 10.7s, avg = 14.0s, dev = 3.2s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 37.9s Stats over 5 runs: max = 37.9s, min = 35.3s, avg = 36.3s, dev = 1.1s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 30.5s Stats over 5 runs: max = 30.5s, min = 25.1s, avg = 27.6s, dev = 2.1s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 22.3s Stats over 5 runs: max = 22.3s, min = 16.5s, avg = 19.2s, dev = 2.2s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 10.4s Stats over 5 runs: max = 10.4s, min = 9.0s, avg = 9.9s, dev = 0.5s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 10.9s Stats over 5 runs: max = 10.9s, min = 9.4s, avg = 10.1s, dev = 0.6s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 14.3s Stats over 5 runs: max = 14.3s, min = 11.6s, avg = 13.1s, dev = 1.0s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 17.3s Stats over 5 runs: max = 17.3s, min = 15.6s, avg = 16.3s, dev = 0.6s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 106.0s Stats over 5 runs: max = 106.0s, min = 16.6s, avg = 49.3s, dev = 30.8s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 100.6s Stats over 5 runs: max = 100.6s, min = 17.4s, avg = 51.5s, dev = 28.0s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 2.8s Stats over 5 runs: max = 2.8s, min = 2.0s, avg = 2.4s, dev = 0.3s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 10.9s Stats over 5 runs: max = 10.9s, min = 9.0s, avg = 10.1s, dev = 0.8s //tensorflow/dtensor/python/tests:multi_mesh_test_cpu PASSED in 11.4s Stats over 5 runs: max = 11.4s, min = 10.4s, avg = 11.0s, dev = 0.3s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 14.1s Stats over 5 runs: max = 14.1s, min = 12.7s, avg = 13.5s, dev = 0.5s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 14.9s Stats over 5 runs: max = 14.9s, min = 13.9s, avg = 14.5s, dev = 0.4s //tensorflow/python/distribute:vars_test_2gpu PASSED in 44.3s Stats over 5 runs: max = 44.3s, min = 37.0s, avg = 41.6s, dev = 2.5s //tensorflow/python/distribute:vars_test_cpu PASSED in 22.8s Stats over 5 runs: max = 22.8s, min = 21.6s, avg = 22.2s, dev = 0.5s //tensorflow/python/eager:device_placement_test_cpu PASSED in 11.8s Stats over 5 runs: max = 11.8s, min = 9.8s, avg = 10.7s, dev = 0.7s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 105.1s Stats over 5 runs: max = 105.1s, min = 19.7s, avg = 51.7s, dev = 28.5s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 17.7s Stats over 5 runs: max = 17.7s, min = 11.6s, avg = 14.9s, dev = 2.6s //tensorflow/python/grappler:cluster_test_cpu PASSED in 9.8s Stats over 5 runs: max = 9.8s, min = 8.9s, avg = 9.4s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 49.4s Stats over 5 runs: max = 49.4s, min = 35.3s, avg = 42.7s, dev = 4.7s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 54.8s Stats over 5 runs: max = 54.8s, min = 51.9s, avg = 52.9s, dev = 1.0s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 93.4s Stats over 5 runs: max = 93.4s, min = 89.1s, avg = 90.7s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 45.9s Stats over 5 runs: max = 45.9s, min = 43.1s, avg = 44.7s, dev = 1.2s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 69.1s Stats over 5 runs: max = 69.1s, min = 64.7s, avg = 66.3s, dev = 1.6s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 53.6s Stats over 5 runs: max = 53.6s, min = 51.6s, avg = 52.6s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 116.8s Stats over 5 runs: max = 116.8s, min = 102.3s, avg = 105.7s, dev = 5.6s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 45.4s Stats over 5 runs: max = 45.4s, min = 43.0s, avg = 44.2s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 56.4s Stats over 5 runs: max = 56.4s, min = 54.8s, avg = 55.4s, dev = 0.6s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 73.0s Stats over 5 runs: max = 73.0s, min = 68.9s, avg = 70.4s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 11.0s Stats over 5 runs: max = 11.0s, min = 10.5s, avg = 10.8s, dev = 0.2s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 43.6s Stats over 5 runs: max = 43.6s, min = 42.2s, avg = 43.0s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 134.8s Stats over 5 runs: max = 134.8s, min = 8.7s, avg = 34.6s, dev = 50.1s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 15.3s Stats over 5 runs: max = 15.3s, min = 9.1s, avg = 11.1s, dev = 2.3s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 16.3s Stats over 5 runs: max = 16.3s, min = 9.0s, avg = 10.9s, dev = 2.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 27.0s Stats over 5 runs: max = 27.0s, min = 8.5s, avg = 13.3s, dev = 7.0s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 87.4s Stats over 5 runs: max = 87.4s, min = 50.1s, avg = 66.2s, dev = 12.8s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 17.3s Stats over 6 runs: max = 17.3s, min = 12.5s, avg = 15.1s, dev = 1.6s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 28.6s Stats over 6 runs: max = 28.6s, min = 19.4s, avg = 25.2s, dev = 2.8s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 27.3s Stats over 6 runs: max = 27.3s, min = 9.7s, avg = 17.2s, dev = 7.3s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 68.8s Stats over 6 runs: max = 68.8s, min = 9.9s, avg = 22.8s, dev = 20.7s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 53.2s Stats over 6 runs: max = 53.2s, min = 30.9s, avg = 41.3s, dev = 7.0s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 16.0s Stats over 7 runs: max = 16.0s, min = 11.8s, avg = 13.3s, dev = 1.5s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 11.3s Stats over 8 runs: max = 11.3s, min = 10.3s, avg = 10.6s, dev = 0.4s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 52.8s Stats over 8 runs: max = 52.8s, min = 9.3s, avg = 22.7s, dev = 14.8s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 71.3s Stats over 8 runs: max = 71.3s, min = 11.2s, avg = 32.0s, dev = 21.3s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 21.6s Stats over 8 runs: max = 21.6s, min = 12.7s, avg = 15.7s, dev = 2.8s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 16.7s Stats over 8 runs: max = 16.7s, min = 4.6s, avg = 9.5s, dev = 4.2s //tensorflow/dtensor/python/tests:input_util_test PASSED in 30.1s Stats over 8 runs: max = 30.1s, min = 19.3s, avg = 25.1s, dev = 3.4s //tensorflow/dtensor/python/tests:save_restore_v2_test_cpu PASSED in 19.8s Stats over 8 runs: max = 19.8s, min = 9.2s, avg = 12.9s, dev = 4.0s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 34.6s Stats over 8 runs: max = 34.6s, min = 14.0s, avg = 21.3s, dev = 7.6s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 35.5s Stats over 8 runs: max = 35.5s, min = 15.0s, avg = 25.8s, dev = 7.1s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 40.8s Stats over 8 runs: max = 40.8s, min = 10.4s, avg = 24.9s, dev = 12.9s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 28.8s Stats over 8 runs: max = 28.8s, min = 9.3s, avg = 16.2s, dev = 7.9s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 27.3s Stats over 8 runs: max = 27.3s, min = 10.8s, avg = 17.0s, dev = 6.3s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 18.5s Stats over 8 runs: max = 18.5s, min = 10.4s, avg = 13.5s, dev = 3.4s //tensorflow/python/data/kernel_tests:batch_test PASSED in 33.6s Stats over 8 runs: max = 33.6s, min = 24.3s, avg = 28.1s, dev = 2.9s //tensorflow/python/data/kernel_tests:filter_test PASSED in 25.7s Stats over 8 runs: max = 25.7s, min = 17.7s, avg = 20.8s, dev = 2.2s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 25.9s Stats over 8 runs: max = 25.9s, min = 16.1s, avg = 19.9s, dev = 3.4s //tensorflow/python/data/kernel_tests:shard_test PASSED in 32.8s Stats over 8 runs: max = 32.8s, min = 22.3s, avg = 27.0s, dev = 3.1s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 62.6s Stats over 8 runs: max = 62.6s, min = 31.7s, avg = 37.0s, dev = 9.7s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 30.1s Stats over 8 runs: max = 30.1s, min = 20.6s, avg = 25.1s, dev = 2.7s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 74.1s Stats over 8 runs: max = 74.1s, min = 14.5s, avg = 33.1s, dev = 23.8s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 58.2s Stats over 8 runs: max = 58.2s, min = 36.2s, avg = 48.3s, dev = 7.7s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 139.0s Stats over 8 runs: max = 139.0s, min = 100.4s, avg = 121.2s, dev = 12.0s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 90.0s Stats over 8 runs: max = 90.0s, min = 74.4s, avg = 81.6s, dev = 6.7s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 66.9s Stats over 8 runs: max = 66.9s, min = 9.5s, avg = 19.1s, dev = 18.8s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 10.4s Stats over 8 runs: max = 10.4s, min = 8.7s, avg = 9.6s, dev = 0.6s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 53.5s Stats over 8 runs: max = 53.5s, min = 30.6s, avg = 39.7s, dev = 8.0s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 25.2s Stats over 8 runs: max = 25.2s, min = 14.9s, avg = 17.8s, dev = 3.1s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 9.9s Stats over 10 runs: max = 9.9s, min = 9.4s, avg = 9.8s, dev = 0.2s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 9.5s Stats over 10 runs: max = 9.5s, min = 5.0s, avg = 8.0s, dev = 1.8s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 15.5s Stats over 10 runs: max = 15.5s, min = 8.8s, avg = 12.6s, dev = 2.3s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 17.4s Stats over 10 runs: max = 17.4s, min = 10.9s, avg = 14.3s, dev = 1.9s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 86.7s Stats over 10 runs: max = 86.7s, min = 44.5s, avg = 61.6s, dev = 13.3s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 93.8s Stats over 10 runs: max = 93.8s, min = 48.6s, avg = 72.8s, dev = 15.7s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 27.1s Stats over 10 runs: max = 27.1s, min = 8.3s, avg = 14.3s, dev = 5.0s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 21.5s Stats over 10 runs: max = 21.5s, min = 10.4s, avg = 15.7s, dev = 4.2s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 27.5s Stats over 10 runs: max = 27.5s, min = 13.1s, avg = 20.0s, dev = 5.1s //tensorflow/python/framework:function_test_cpu PASSED in 59.9s Stats over 10 runs: max = 59.9s, min = 6.3s, avg = 14.5s, dev = 15.3s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 15.4s Stats over 10 runs: max = 15.4s, min = 9.1s, avg = 12.7s, dev = 2.2s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 10.3s Stats over 10 runs: max = 10.3s, min = 6.3s, avg = 8.5s, dev = 1.2s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 13.6s Stats over 10 runs: max = 13.6s, min = 9.5s, avg = 11.3s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 93.5s Stats over 10 runs: max = 93.5s, min = 84.4s, avg = 87.2s, dev = 2.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 76.7s Stats over 10 runs: max = 76.7s, min = 13.6s, avg = 43.5s, dev = 19.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 11.0s Stats over 10 runs: max = 11.0s, min = 8.4s, avg = 10.1s, dev = 1.0s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 14.9s Stats over 10 runs: max = 14.9s, min = 9.5s, avg = 12.8s, dev = 1.6s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 29.3s Stats over 10 runs: max = 29.3s, min = 7.1s, avg = 17.3s, dev = 8.5s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 26.1s Stats over 10 runs: max = 26.1s, min = 9.0s, avg = 12.9s, dev = 5.9s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 18.7s Stats over 10 runs: max = 18.7s, min = 12.4s, avg = 14.6s, dev = 1.7s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 12.3s Stats over 10 runs: max = 12.3s, min = 10.7s, avg = 11.4s, dev = 0.5s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 80.1s Stats over 10 runs: max = 80.1s, min = 18.7s, avg = 48.6s, dev = 29.0s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 53.8s Stats over 10 runs: max = 53.8s, min = 12.0s, avg = 19.9s, dev = 11.7s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 13.0s Stats over 10 runs: max = 13.0s, min = 9.7s, avg = 10.9s, dev = 1.0s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 119.4s Stats over 10 runs: max = 119.4s, min = 110.2s, avg = 114.7s, dev = 3.0s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 21.7s Stats over 10 runs: max = 21.7s, min = 17.7s, avg = 20.0s, dev = 1.1s //tensorflow/python/saved_model:load_test_cpu PASSED in 75.7s Stats over 10 runs: max = 75.7s, min = 49.5s, avg = 55.4s, dev = 7.2s //tensorflow/python/distribute/failure_handling:failure_handler_test FLAKY, failed in 2 out of 10 in 64.4s Stats over 10 runs: max = 64.4s, min = 25.1s, avg = 52.2s, dev = 13.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_5_of_8/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_1_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:fft_test_cpu PASSED in 25.5s Stats over 12 runs: max = 25.5s, min = 12.4s, avg = 19.3s, dev = 4.3s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 19.9s Stats over 12 runs: max = 19.9s, min = 8.8s, avg = 13.6s, dev = 3.7s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 21.3s Stats over 12 runs: max = 21.3s, min = 6.2s, avg = 11.0s, dev = 4.4s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 15.5s Stats over 12 runs: max = 15.5s, min = 5.3s, avg = 8.6s, dev = 2.9s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 19.5s Stats over 12 runs: max = 19.5s, min = 15.1s, avg = 17.3s, dev = 1.5s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 26.4s Stats over 12 runs: max = 26.4s, min = 22.5s, avg = 24.4s, dev = 1.1s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 226.4s Stats over 12 runs: max = 226.4s, min = 15.3s, avg = 55.5s, dev = 57.2s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 232.6s Stats over 12 runs: max = 232.6s, min = 16.9s, avg = 55.2s, dev = 59.4s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 24.6s Stats over 15 runs: max = 24.6s, min = 12.6s, avg = 17.9s, dev = 3.5s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 106.7s Stats over 15 runs: max = 106.7s, min = 100.2s, avg = 104.2s, dev = 2.0s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 54.9s Stats over 15 runs: max = 54.9s, min = 9.4s, avg = 19.0s, dev = 10.9s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 14.3s Stats over 16 runs: max = 14.3s, min = 3.6s, avg = 9.6s, dev = 2.8s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 44.7s Stats over 16 runs: max = 44.7s, min = 14.5s, avg = 34.7s, dev = 6.3s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 78.5s Stats over 16 runs: max = 78.5s, min = 50.5s, avg = 61.0s, dev = 6.9s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 34.9s Stats over 16 runs: max = 34.9s, min = 29.5s, avg = 31.1s, dev = 1.2s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 16.3s Stats over 16 runs: max = 16.3s, min = 14.3s, avg = 15.2s, dev = 0.6s //tensorflow/python/ops:image_ops_test_cpu PASSED in 26.6s Stats over 16 runs: max = 26.6s, min = 14.4s, avg = 18.4s, dev = 3.1s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 112.3s Stats over 17 runs: max = 112.3s, min = 25.8s, avg = 47.7s, dev = 24.0s //tensorflow/python/data/kernel_tests:map_test PASSED in 39.2s Stats over 19 runs: max = 39.2s, min = 10.0s, avg = 22.7s, dev = 7.0s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 10.3s Stats over 20 runs: max = 10.3s, min = 4.4s, avg = 7.1s, dev = 2.0s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 10.1s Stats over 20 runs: max = 10.1s, min = 3.9s, avg = 6.9s, dev = 2.0s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 17.2s Stats over 20 runs: max = 17.2s, min = 3.9s, avg = 7.2s, dev = 3.1s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 11.9s Stats over 20 runs: max = 11.9s, min = 4.3s, avg = 8.4s, dev = 2.0s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 11.3s Stats over 20 runs: max = 11.3s, min = 5.5s, avg = 7.8s, dev = 1.9s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 32.3s Stats over 20 runs: max = 32.3s, min = 5.6s, avg = 14.9s, dev = 9.6s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 46.4s Stats over 20 runs: max = 46.4s, min = 4.2s, avg = 12.6s, dev = 12.0s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 17.0s Stats over 20 runs: max = 17.0s, min = 12.8s, avg = 15.2s, dev = 1.1s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 143.4s Stats over 20 runs: max = 143.4s, min = 132.4s, avg = 138.1s, dev = 2.6s //tensorflow/python/kernel_tests:metrics_test PASSED in 41.8s Stats over 20 runs: max = 41.8s, min = 9.0s, avg = 21.0s, dev = 9.4s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 9.5s Stats over 20 runs: max = 9.5s, min = 5.1s, avg = 7.7s, dev = 1.2s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 17.5s Stats over 20 runs: max = 17.5s, min = 4.3s, avg = 8.9s, dev = 3.2s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 56.1s Stats over 20 runs: max = 56.1s, min = 7.0s, avg = 18.7s, dev = 16.4s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 123.7s Stats over 20 runs: max = 123.7s, min = 50.5s, avg = 75.6s, dev = 22.7s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 11.3s Stats over 20 runs: max = 11.3s, min = 6.2s, avg = 8.3s, dev = 1.7s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 16.1s Stats over 20 runs: max = 16.1s, min = 5.7s, avg = 10.9s, dev = 2.9s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 156.4s Stats over 20 runs: max = 156.4s, min = 39.4s, avg = 95.1s, dev = 35.2s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 25.9s Stats over 20 runs: max = 25.9s, min = 8.3s, avg = 13.9s, dev = 5.5s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 21.7s Stats over 20 runs: max = 21.7s, min = 5.8s, avg = 13.4s, dev = 5.6s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 21.2s Stats over 20 runs: max = 21.2s, min = 14.9s, avg = 18.9s, dev = 1.9s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 79.4s Stats over 20 runs: max = 79.4s, min = 10.2s, avg = 34.2s, dev = 23.2s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 22.2s Stats over 20 runs: max = 22.2s, min = 9.0s, avg = 11.7s, dev = 2.9s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 30.3s Stats over 24 runs: max = 30.3s, min = 7.8s, avg = 18.8s, dev = 6.6s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 37.6s Stats over 24 runs: max = 37.6s, min = 5.3s, avg = 23.5s, dev = 12.7s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 33.9s Stats over 25 runs: max = 33.9s, min = 26.1s, avg = 30.1s, dev = 2.3s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 15.6s Stats over 30 runs: max = 15.6s, min = 4.3s, avg = 8.5s, dev = 2.9s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 33.5s Stats over 32 runs: max = 33.5s, min = 6.0s, avg = 13.5s, dev = 7.0s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 26.2s Stats over 32 runs: max = 26.2s, min = 5.2s, avg = 13.9s, dev = 4.8s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.6s Stats over 48 runs: max = 1.6s, min = 0.4s, avg = 0.6s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 58.9s Stats over 50 runs: max = 58.9s, min = 31.7s, avg = 42.9s, dev = 8.5s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 16.9s Stats over 50 runs: max = 16.9s, min = 4.1s, avg = 10.3s, dev = 3.1s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 23.8s Stats over 50 runs: max = 23.8s, min = 4.6s, avg = 13.4s, dev = 4.3s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 51.2s Stats over 50 runs: max = 51.2s, min = 31.6s, avg = 40.6s, dev = 4.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 16.6s Stats over 50 runs: max = 16.6s, min = 5.5s, avg = 9.9s, dev = 2.7s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 25.4s Stats over 50 runs: max = 25.4s, min = 7.0s, avg = 13.8s, dev = 4.7s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 14.1s Stats over 50 runs: max = 14.1s, min = 3.9s, avg = 5.6s, dev = 1.9s Executed 3050 out of 3050 tests: 3049 tests pass and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.