==================== Test output for //tensorflow/compiler/xla/pjrt/distributed:client_server_test: [==========] Running 16 tests from 1 test suite. [----------] Global test environment set-up. [----------] 16 tests from ClientServerTest [ RUN ] ClientServerTest.ConnectAndShutdownAreBarriers 2023-08-30 00:10:29.281781: I tensorflow/compiler/xla/pjrt/distributed/service.cc:119] Experimental coordination service is enabled. 2023-08-30 00:10:29.329744: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:jax_worker/replica:0/task:1 has connected to coordination service. Incarnation: 7111866353349301924 2023-08-30 00:10:29.336149: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:jax_worker/replica:0/task:2 has connected to coordination service. Incarnation: 18079729612147800276 2023-08-30 00:10:29.336681: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. 2023-08-30 00:10:29.346276: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:jax_worker/replica:0/task:0 has connected to coordination service. Incarnation: 15321680584333416718 2023-08-30 00:10:29.346534: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. 2023-08-30 00:10:29.356352: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. 2023-08-30 00:10:31.711258: I tensorflow/compiler/xla/pjrt/distributed/client.cc:134] Failed to connect to distributed JAX controller: INTERNAL: Barrier failed from a task error. Barrier Id: PjRT_Client_Connect, Task: /job:jax_worker/replica:0/task:2 Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Barrier: :{"created":"@1693354231.283839033","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Barrier failed from a task error. Barrier Id: PjRT_Client_Connect, Task: /job:jax_worker/replica:0/task:2","grpc_status":13} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-08-30 00:10:31.711341: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:472] Coordination agent has initiated Shutdown(). 2023-08-30 00:10:31.711557: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:990] /job:jax_worker/replica:0/task:2 has been set to ERROR in coordination service: UNAVAILABLE: Task /job:jax_worker/replica:0/task:2 heartbeat timeout. This indicates that the remote task has failed, got preempted, or crashed unexpectedly. Check the task logs for an earlier error to debug further. [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-08-30 00:10:31.711604: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:418] Stopping coordination service as the following tasks are unhealthy (stopped sending heartbeats): /job:jax_worker/replica:0/task:2 Check the task logs for an earlier error to debug further. 2023-08-30 00:10:31.711648: I tensorflow/compiler/xla/pjrt/distributed/client.cc:134] Failed to connect to distributed JAX controller: INTERNAL: Barrier failed from a task error. Barrier Id: PjRT_Client_Connect, Task: /job:jax_worker/replica:0/task:2 Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Barrier: :{"created":"@1693354231.284290050","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Barrier failed from a task error. Barrier Id: PjRT_Client_Connect, Task: /job:jax_worker/replica:0/task:2","grpc_status":13} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-08-30 00:10:31.711728: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:472] Coordination agent has initiated Shutdown(). 2023-08-30 00:10:31.711975: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:1167] Shutdown barrier in coordination service has failed: INVALID_ARGUMENT: A non-participating task (/job:jax_worker/replica:0/task:1) called the barrier: Shutdown::10243288331382099046 [type.googleapis.com/tensorflow.CoordinationServiceError=''] This suggests that the workers are out of sync. Either at least one worker is too fast in its execution / crashed early or too slow / hanging. Check the logs for an earlier error to identify the root cause. 2023-08-30 00:10:31.712013: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:769] Coordination agent is set to ERROR: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:1. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause. Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat: :{"created":"@1693354231.711840997","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:1. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-08-30 00:10:31.712049: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:769] Coordination agent is set to ERROR: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause. Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat: :{"created":"@1693354231.711859754","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-08-30 00:10:31.712080: E tensorflow/compiler/xla/pjrt/distributed/client.cc:96] Coordination service agent in error status: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:1. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause. Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat: :{"created":"@1693354231.711840997","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:1. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-08-30 00:10:31.712130: F ./tensorflow/compiler/xla/pjrt/distributed/client.h:77] Terminating process because the coordinator detected missing heartbeats. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. Status: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:1. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause. Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat: :{"created":"@1693354231.711840997","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:1. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] *** Received signal 6 *** *** BEGIN MANGLED STACK TRACE *** 2023-08-30 00:10:31.712352: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:493] Failed to disconnect from coordination service with status: INVALID_ARGUMENT: Unexpected disconnect request with task_name=/job:jax_worker/replica:0/task:0 Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/ShutdownTask: :{"created":"@1693354231.712240314","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected disconnect request with task_name=/job:jax_worker/replica:0/task:0","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] Proceeding with agent shutdown anyway. This is usually caused by an earlier error during execution. Check the logs (this task or the leader) for an earlier error to debug further. 2023-08-30 00:10:31.712450: E tensorflow/compiler/xla/pjrt/distributed/client.cc:96] Coordination service agent in error status: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause. Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat: :{"created":"@1693354231.711859754","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] 2023-08-30 00:10:31.712485: F ./tensorflow/compiler/xla/pjrt/distributed/client.h:77] Terminating process because the coordinator detected missing heartbeats. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. Status: INVALID_ARGUMENT: Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause. Additional GRPC error information from remote target unknown_target_for_coordination_leader while calling /tensorflow.CoordinationService/Heartbeat: :{"created":"@1693354231.711859754","description":"Error received from peer inproc","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unexpected heartbeat request from task: /job:jax_worker/replica:0/task:0. This usually implies an earlier error that caused coordination service to shut down before the workers disconnect. Check the task leader's logs for an earlier error to debug the root cause.","grpc_status":3} [type.googleapis.com/tensorflow.CoordinationServiceError=''] ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:gce_failure_handler_test (shard 7 of 8): Running tests under Python 3.11.1: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 43181 I0830 00:08:44.218792 281472882735808 test_util.py:3820] Using local port 43181 INFO:tensorflow:Using local port 33795 I0830 00:08:44.219630 281472882735808 test_util.py:3820] Using local port 33795 INFO:tensorflow:Using local port 42355 I0830 00:08:44.220033 281472882735808 test_util.py:3820] Using local port 42355 INFO:tensorflow:Using local port 34491 I0830 00:08:44.220424 281472882735808 test_util.py:3820] Using local port 34491 INFO:tensorflow:Cluster starting. I0830 00:08:48.686610 281472882735808 gce_failure_handler_test.py:317] Cluster starting. [worker-1]: I0830 00:08:48.738306 281473157069504 multi_process_runner.py:840] Subprocess with PID 1669747 (worker, 1) is now being started. [worker-1]: I0830 00:08:48.738755 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-1]: 2023-08-30 00:08:48.846728: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33795 [worker-0]: I0830 00:08:48.952262 281473157069504 multi_process_runner.py:840] Subprocess with PID 1669743 (worker, 0) is now being started. [worker-2]: I0830 00:08:48.951384 281473157069504 multi_process_runner.py:840] Subprocess with PID 1669751 (worker, 2) is now being started. [worker-3]: I0830 00:08:48.954674 281473157069504 multi_process_runner.py:840] Subprocess with PID 1669757 (worker, 3) is now being started. [worker-0]: I0830 00:08:48.952747 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:08:48.951961 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:08:48.955189 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2023-08-30 00:08:49.156669: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42355 [worker-0]: 2023-08-30 00:08:49.166958: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:43181 [worker-3]: 2023-08-30 00:08:49.191511: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34491 [worker-0]: 2023-08-30 00:08:49.236221: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 11357196870979570477 [worker-0]: 2023-08-30 00:08:49.236350: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 3896486815786029983 [worker-2]: 2023-08-30 00:08:49.236781: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-08-30 00:08:49.237237: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:08:49.245499: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 17191579846194876974 [worker-0]: 2023-08-30 00:08:49.245851: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:08:49.896767: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 6185204672415335478 [worker-1]: 2023-08-30 00:08:49.956183: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0830 00:08:49.959037 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:08:49.977172 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:08:49.967681 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0830 00:08:49.968374 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0830 00:08:50.013483 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:08:50.026777 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:08:50.027049 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-2]: I0830 00:08:50.085599 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: I0830 00:08:50.073183 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0830 00:08:50.073708 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:08:50.073944 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0830 00:08:50.097195 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:08:50.097559 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0830 00:08:50.161705 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:08:50.162250 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:08:50.162492 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0830 00:08:50.262545 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0830 00:08:50.263285 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0830 00:08:50.263703 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0830 00:08:50.264018 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0830 00:08:50.264222 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0830 00:08:50.360052 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0830 00:08:50.349624 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0830 00:08:50.360799 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0830 00:08:50.350392 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: I0830 00:08:50.361238 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0830 00:08:50.350869 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: Instructions for updating: [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-1]: W0830 00:08:50.361569 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-0]: W0830 00:08:50.351207 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-1]: INFO:tensorflow:Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: I0830 00:08:50.361778 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: I0830 00:08:50.364023 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start training at 0 [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0830 00:08:50.351419 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: I0830 00:08:50.364719 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0830 00:08:50.365216 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0830 00:08:50.365566 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0830 00:08:50.365776 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:50.424495 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:50.464346 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:50.473457 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:50.514222 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:50.610497 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:50.613344 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:50.630701 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:50.621044 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:50.695724 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:50.695856 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:50.721442 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:50.724416 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:50.832775 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:50.833139 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:50.862656 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:50.870375 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:50.948998 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:50.951879 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:50.964693 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:50.970435 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:08:51.036875 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:08:51.037259 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:08:51.037594 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f100> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:08:51.045019 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f100> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:51.046229 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:51.047171 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:51.053837 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:51.046548 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ed40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:08:51.117253 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ed40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0830 00:08:51.117721 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23e3e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:08:51.119405 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23e3e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0830 00:08:51.119754 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f600> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:08:51.118961 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f600> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:08:51.117906 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-2]: I0830 00:08:51.119316 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: I0830 00:08:51.118276 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:51.127200 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:51.131240 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:51.133310 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:51.127949 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:51.196939 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:51.197073 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:51.201752 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:51.202649 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:51.276046 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:51.280612 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:51.286317 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:51.290561 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:51.419511 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:51.407556 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:51.444039 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:51.440060 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:51.577280 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:51.598846 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:51.603448 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:51.606565 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:51.687118 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:51.687280 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:51.700388 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:51.700488 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0830 00:08:52.108376 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0830 00:08:52.115420 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-1]: I0830 00:08:52.125410 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0830 00:08:52.120806 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:52.131536 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:52.140323 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:52.150175 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:52.310050 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:52.385982 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:52.388576 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:52.388331 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:52.410045 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:52.521743 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:52.550399 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:52.540017 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:52.580108 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:52.710891 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:52.720742 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:52.740780 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:52.760291 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:52.864278 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:52.870369 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:52.889992 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:52.899538 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:52.965388 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:52.973848 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:53.010888 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:53.010908 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0830 00:08:53.072073 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:53.078022 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: I0830 00:08:53.083006 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0830 00:08:53.085994 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0830 00:08:53.091735 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:53.100962 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:53.107119 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:53.134203 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:53.246169 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:53.266608 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:53.272237 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:53.275075 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:53.334089 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:53.334691 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:53.369782 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Termination notice available. [worker-3]: I0830 00:08:53.430332 281456706187744 gce_failure_handler_test.py:142] Termination notice available. [worker-3]: INFO:tensorflow:Member 3 has received termination notice. [worker-3]: I0830 00:08:53.436869 281456706187744 failure_handling.py:710] Member 3 has received termination notice. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:53.442171 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-3]: I0830 00:08:53.502174 281473157069504 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-0]: I0830 00:08:53.504387 281453132509664 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: I0830 00:08:53.504452 281448434889184 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-3]: I0830 00:08:53.504578 281456723096032 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-3]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 22 [worker-3]: I0830 00:08:53.504901 281473157069504 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 22 [worker-3]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-3]: I0830 00:08:53.507590 281473157069504 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-3]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-3]: I0830 00:08:53.508264 281473157069504 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-3]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-3]: I0830 00:08:53.508836 281473157069504 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-3]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-3]: I0830 00:08:53.509391 281473157069504 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-1]: I0830 00:08:53.504340 281447746892256 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:53.515039 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:53.528349 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:53.530227 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:53.529029 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0830 00:08:53.588066 281473157069504 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0830 00:08:53.588662 281473157069504 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0830 00:08:53.588465 281473157069504 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0830 00:08:53.588398 281473157069504 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/workertemp_3/ [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/workertemp_1/ [worker-1]: I0830 00:08:53.758549 281473157069504 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/workertemp_1/ [worker-3]: I0830 00:08:53.757899 281473157069504 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/workertemp_3/ [worker-3]: INFO:tensorflow:Shut down watcher for one's own termination signal [worker-3]: I0830 00:08:53.758288 281473157069504 failure_handling.py:737] Shut down watcher for one's own termination signal [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0830 00:08:53.760000 281473157069504 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0830 00:08:53.760183 281473157069504 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/ [worker-0]: I0830 00:08:53.768876 281473157069504 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/ [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/workertemp_2/ [worker-2]: I0830 00:08:53.771734 281473157069504 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/284377733e18a9a7ee6d6d7363a8b705wb__51dg/tmp0vdkg0ix/fh_ckpt/workertemp_2/ [worker-2]: INFO:tensorflow:Shut down watcher for one's own termination signal [worker-1]: INFO:tensorflow:Shut down watcher for one's own termination signal [worker-0]: INFO:tensorflow:Shut down watcher for one's own termination signal [worker-2]: I0830 00:08:54.313684 281473157069504 failure_handling.py:737] Shut down watcher for one's own termination signal [worker-1]: I0830 00:08:54.406464 281473157069504 failure_handling.py:737] Shut down watcher for one's own termination signal [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0830 00:08:54.455959 281473157069504 failure_handling.py:737] Shut down watcher for one's own termination signal [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: I0830 00:08:54.314944 281473157069504 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0830 00:08:54.408079 281473157069504 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: I0830 00:08:54.457744 281473157069504 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: I0830 00:08:54.315118 281473157069504 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0830 00:08:54.457907 281473157069504 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0830 00:08:54.408241 281473157069504 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0830 00:08:55.777338 281472882735808 gce_failure_handler_test.py:323] restarting workers [worker-0]: I0830 00:08:55.827412 281473157069504 multi_process_runner.py:840] Subprocess with PID 1675833 (worker, 0) is now being started. [worker-0]: I0830 00:08:55.828002 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' INFO:tensorflow:workers restarted I0830 00:08:55.853461 281472882735808 gce_failure_handler_test.py:327] workers restarted [worker-1]: I0830 00:08:55.857709 281473157069504 multi_process_runner.py:840] Subprocess with PID 1675836 (worker, 1) is now being started. [worker-1]: I0830 00:08:55.858268 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:08:55.870207 281473157069504 multi_process_runner.py:840] Subprocess with PID 1675854 (worker, 2) is now being started. [worker-2]: I0830 00:08:55.870756 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:08:55.892866 281473157069504 multi_process_runner.py:840] Subprocess with PID 1675956 (worker, 3) is now being started. [worker-3]: I0830 00:08:55.893374 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:43181", "localhost:33795", "localhost:42355", "localhost:34491"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-08-30 00:08:55.895013: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:43181 [worker-2]: 2023-08-30 00:08:55.906835: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42355 [worker-2]: 2023-08-30 00:08:55.970442: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:08:55.969859: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 3067628660080044464 [worker-0]: 2023-08-30 00:08:55.970144: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 6714342785008972933 [worker-0]: 2023-08-30 00:08:55.973735: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-08-30 00:08:56.037255: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34491 [worker-0]: 2023-08-30 00:08:56.040480: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 5305289115365854955 [worker-3]: 2023-08-30 00:08:56.056155: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2023-08-30 00:08:56.327006: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33795 [worker-0]: 2023-08-30 00:08:56.336903: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2379770447643876164 [worker-1]: 2023-08-30 00:08:56.337189: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0830 00:08:56.377007 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0830 00:08:56.396982 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:08:56.417089 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:08:56.403237 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0830 00:08:56.526890 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:08:56.527455 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:08:56.527699 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: I0830 00:08:56.528071 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:08:56.541051 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: I0830 00:08:56.528601 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Check health not enabled. [worker-2]: I0830 00:08:56.528834 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:08:56.541601 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:08:56.541834 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0830 00:08:56.673704 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0830 00:08:56.674358 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:08:56.674597 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:43181', 'localhost:33795', 'localhost:42355', 'localhost:34491']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0830 00:08:56.890722 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0830 00:08:56.892946 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0830 00:08:56.898792 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0830 00:08:56.906831 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0830 00:08:56.907928 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0830 00:08:56.916585 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0830 00:08:56.922118 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0830 00:08:56.922555 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 22 [worker-3]: I0830 00:08:56.922772 281473157069504 gce_failure_handler_test.py:194] Start training at 22 [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0830 00:08:56.935310 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0830 00:08:56.936311 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0830 00:08:56.936739 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 22 [worker-0]: I0830 00:08:56.936959 281473157069504 gce_failure_handler_test.py:194] Start training at 22 [worker-3]: INFO:tensorflow:['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-3]: I0830 00:08:56.933618 281473157069504 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0830 00:08:56.943799 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0830 00:08:56.957526 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: I0830 00:08:56.956224 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-2]: W0830 00:08:56.957962 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-1]: W0830 00:08:56.956723 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-2]: INFO:tensorflow:Start training at 22 [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: I0830 00:08:56.958173 281473157069504 gce_failure_handler_test.py:194] Start training at 22 [worker-1]: INFO:tensorflow:Start training at 22 [worker-1]: I0830 00:08:56.956954 281473157069504 gce_failure_handler_test.py:194] Start training at 22 [worker-0]: INFO:tensorflow:['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-0]: I0830 00:08:56.957654 281473157069504 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-1]: INFO:tensorflow:['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-1]: I0830 00:08:56.992255 281473157069504 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-2]: INFO:tensorflow:['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-2]: I0830 00:08:56.997609 281473157069504 gce_failure_handler_test.py:203] ['workertemp_2', 'workertemp_3', 'workertemp_1', 'ckpt-1.data-00000-of-00001', 'ckpt-1.index', 'checkpoint'] [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:57.098640 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:57.170613 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:57.203859 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:57.322965 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:57.395280 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:57.420818 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:57.420679 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:57.477198 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0830 00:08:57.749663 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:08:57.756444 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0830 00:08:57.756356 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: I0830 00:08:57.756076 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:57.767112 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:57.777164 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:57.781444 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:57.781494 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:57.905953 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:57.941071 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:57.931387 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:57.959893 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:58.019880 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:58.019918 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:58.019834 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:58.037090 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23df80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f22c9a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:08:58.133915 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f22c9a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:08:58.127369 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23df80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f1dccc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:08:58.139820 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f1dccc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:58.150219 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:58.167931 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:58.190835 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f380> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:08:58.207197 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f380> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:58.261094 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:08:58.383042 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23eac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f22f920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:08:58.390355 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f22f920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:08:58.389731 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23eac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df420> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:08:58.399927 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df420> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:58.411243 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:58.431452 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:58.435927 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:58.511365 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:08:58.670486 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:08:58.674637 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:08:58.672730 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:08:58.680754 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: I0830 00:08:58.796703 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0830 00:08:58.798542 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0830 00:08:58.796960 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0830 00:08:58.798829 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0830 00:08:58.797245 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0830 00:08:58.797021 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0830 00:08:58.803703 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0830 00:08:58.804013 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0830 00:08:58.812060 281473157069504 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0830 00:08:58.826328 281473157069504 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:Shut down watcher for one's own termination signal [worker-3]: I0830 00:08:58.966352 281473157069504 failure_handling.py:737] Shut down watcher for one's own termination signal [worker-0]: INFO:tensorflow:Shut down watcher for one's own termination signal [worker-0]: I0830 00:08:58.974447 281473157069504 failure_handling.py:737] Shut down watcher for one's own termination signal I0830 00:09:01.826785 281472882735808 multi_process_runner.py:646] worker-0 exit code: 0 I0830 00:09:01.827150 281472882735808 multi_process_runner.py:646] worker-1 exit code: 0 I0830 00:09:01.827301 281472882735808 multi_process_runner.py:646] worker-2 exit code: 0 I0830 00:09:01.827437 281472882735808 multi_process_runner.py:646] worker-3 exit code: 0 I0830 00:09:01.829604 281472882735808 multi_process_runner.py:662] Joining log reading threads. I0830 00:09:01.829868 281472882735808 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker): 17.75s I0830 00:09:01.942714 281472882735808 test_util.py:2477] time(__main__.GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker): 17.75s [ OK ] GceFailureHandlingTest.test_basic_run_test_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0830 00:09:02.049985 281472882735808 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0830 00:09:02.050599 281472882735808 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start polling for termination signal. I0830 00:09:02.105506 281472882735808 failure_handling.py:683] Start polling for termination signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0830 00:09:02.106413 281472882735808 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0830 00:09:02.106760 281472882735808 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0830 00:09:02.106962 281472882735808 gce_failure_handler_test.py:194] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xfffed43187c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0830 00:09:02.443679 281472882735808 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xfffed43187c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffed4318680> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0830 00:09:02.471017 281472882735808 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xfffed4318680> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0830 00:09:02.471465 281472882735808 gce_failure_handler_test.py:192] epoch 0 finished INFO:tensorflow:epoch 1 finished I0830 00:09:02.654050 281472882735808 gce_failure_handler_test.py:192] epoch 1 finished INFO:tensorflow:epoch 2 finished I0830 00:09:02.822690 281472882735808 gce_failure_handler_test.py:192] epoch 2 finished INFO:tensorflow:epoch 3 finished I0830 00:09:03.084675 281472882735808 gce_failure_handler_test.py:192] epoch 3 finished INFO:tensorflow:epoch 4 finished I0830 00:09:03.267560 281472882735808 gce_failure_handler_test.py:192] epoch 4 finished INFO:tensorflow:Training finished. I0830 00:09:03.267992 281472882735808 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 1.33s I0830 00:09:03.273340 281472882735808 test_util.py:2477] time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 1.33s [ OK ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0830 00:09:03.286617 281472882735808 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0830 00:09:03.287078 281472882735808 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start polling for termination signal. I0830 00:09:03.302714 281472882735808 failure_handling.py:683] Start polling for termination signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0830 00:09:03.317205 281472882735808 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. INFO:tensorflow:Start training at 0 I0830 00:09:03.317666 281472882735808 gce_failure_handler_test.py:194] Start training at 0 INFO:tensorflow:epoch 0 finished I0830 00:09:03.648052 281472882735808 gce_failure_handler_test.py:192] epoch 0 finished INFO:tensorflow:epoch 1 finished I0830 00:09:03.978284 281472882735808 gce_failure_handler_test.py:192] epoch 1 finished INFO:tensorflow:epoch 2 finished I0830 00:09:04.143185 281472882735808 gce_failure_handler_test.py:192] epoch 2 finished INFO:tensorflow:epoch 3 finished I0830 00:09:04.315646 281472882735808 gce_failure_handler_test.py:192] epoch 3 finished INFO:tensorflow:epoch 4 finished I0830 00:09:04.480475 281472882735808 gce_failure_handler_test.py:192] epoch 4 finished INFO:tensorflow:Training finished. I0830 00:09:04.480890 281472882735808 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 1.21s I0830 00:09:04.485624 281472882735808 test_util.py:2477] time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 1.21s [ OK ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 42597 I0830 00:09:04.490070 281472882735808 test_util.py:3820] Using local port 42597 INFO:tensorflow:Using local port 44773 I0830 00:09:04.490498 281472882735808 test_util.py:3820] Using local port 44773 INFO:tensorflow:Using local port 33797 I0830 00:09:04.490859 281472882735808 test_util.py:3820] Using local port 33797 INFO:tensorflow:Using local port 35777 I0830 00:09:04.491221 281472882735808 test_util.py:3820] Using local port 35777 INFO:tensorflow:Cluster starting. I0830 00:09:04.522089 281472882735808 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0830 00:09:04.720932 281473157069504 multi_process_runner.py:840] Subprocess with PID 1697391 (worker, 0) is now being started. [worker-1]: I0830 00:09:04.725941 281473157069504 multi_process_runner.py:840] Subprocess with PID 1697436 (worker, 1) is now being started. [worker-3]: I0830 00:09:04.728780 281473157069504 multi_process_runner.py:840] Subprocess with PID 1697637 (worker, 3) is now being started. [worker-3]: I0830 00:09:04.729241 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0830 00:09:04.726427 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0830 00:09:04.721422 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:09:04.761542 281473157069504 multi_process_runner.py:840] Subprocess with PID 1697569 (worker, 2) is now being started. [worker-2]: I0830 00:09:04.762024 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: 2023-08-30 00:09:04.786461: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35777 [worker-1]: 2023-08-30 00:09:04.857805: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:44773 [worker-0]: 2023-08-30 00:09:04.860061: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42597 [worker-0]: 2023-08-30 00:09:04.875365: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 10537354463911090363 [worker-3]: 2023-08-30 00:09:04.875836: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:04.891878: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 7672440409579919790 [worker-0]: 2023-08-30 00:09:04.892280: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:04.906314: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 948106896852768673 [worker-2]: 2023-08-30 00:09:04.918711: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33797 [worker-1]: 2023-08-30 00:09:04.926530: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-08-30 00:09:04.938168: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:09:04.940325 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:09:04.940302 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: 2023-08-30 00:09:04.937884: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 18321820976710581837 [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0830 00:09:04.947773 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0830 00:09:04.966943 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0830 00:09:04.997756 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0830 00:09:04.998299 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:09:04.998534 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0830 00:09:05.006524 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:09:05.007287 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:09:05.007544 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0830 00:09:05.021881 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:09:05.022400 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:09:05.022635 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:09:05.025238 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0830 00:09:05.025882 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:09:05.026440 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0830 00:09:05.076571 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0830 00:09:05.076872 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0830 00:09:05.076868 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0830 00:09:05.078105 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: I0830 00:09:05.077519 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0830 00:09:05.078649 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-2]: I0830 00:09:05.078230 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-3]: Traceback (most recent call last): [worker-1]: Traceback (most recent call last): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0830 00:09:05.078794 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: I0830 00:09:05.079624 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: W0830 00:09:05.079195 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-3]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0830 00:09:05.080020 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start training at 0 [worker-1]: Instructions for updating: [worker-3]: I0830 00:09:05.079451 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0830 00:09:05.080369 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: Traceback (most recent call last): [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-0]: I0830 00:09:05.097035 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-2]: self.run() [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: Traceback (most recent call last): [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-2]: if self._termination_watcher_fn(): [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: I0830 00:09:05.098256 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0830 00:09:05.098640 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0830 00:09:05.098851 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0830 00:09:05.098906 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0830 00:09:05.099294 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0830 00:09:05.099511 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:05.199254 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:05.204293 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:05.261846 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:05.272539 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:05.352167 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:05.353697 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:05.353706 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:05.367722 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:05.460008 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:05.471491 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:05.472917 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:05.496357 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:05.607354 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:05.612730 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:05.613243 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:05.632651 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:05.751477 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:05.756007 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:05.761285 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:05.780458 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f242de0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:05.866482 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f242de0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23c5e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:05.867069 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23c5e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f243240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:05.863336 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f243240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:05.873939 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:05.876445 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f2434c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:05.872933 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f2434c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:05.890779 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:05.898778 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243600> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:06.007198 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243600> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0830 00:09:06.007680 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f2434c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:06.011063 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f2434c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0830 00:09:06.011514 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:06.007503 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0830 00:09:06.007912 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243c40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:06.007530 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243c40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0830 00:09:06.007900 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.017714 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.019794 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:06.021259 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.030348 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.099271 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:06.097337 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.099133 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.097553 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:06.160702 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.160640 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.160775 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.221459 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.160745 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.221657 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:06.221136 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.221694 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:06.290912 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.291496 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.291622 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.293129 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:06.509863 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.521123 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.521083 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.548226 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0830 00:09:06.752250 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: I0830 00:09:06.757328 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0830 00:09:06.757472 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0830 00:09:06.763770 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.767075 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.767280 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.773201 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:06.761845 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:06.974549 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:06.979093 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:06.980000 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.000974 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.128590 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:restarting workers [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0830 00:09:09.568806 281472882735808 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:Termination notice available. I0830 00:09:14.328403 281462376624608 gce_failure_handler_test.py:142] Termination notice available. INFO:tensorflow:Member single_worker has received termination notice. I0830 00:09:21.397652 281462376624608 failure_handling.py:701] Member single_worker has received termination notice. Exception ignored in: Traceback (most recent call last): [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 775, in __del__ [worker-2]: I0830 00:09:07.135826 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.129254 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.250696 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 self._stop_poll_termination_signal_thread() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 734, in _stop_poll_termination_signal_thread self._poll_termination_signal_thread.join() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1109, in join raise RuntimeError("cannot join current thread") RuntimeError: cannot join current thread [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.250959 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.312384 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.312387 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.372061 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.372325 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0830 00:09:07.421740 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.431213 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.487817 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.543036 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.597908 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.653837 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.421400 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: I0830 00:09:07.708672 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.150604 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:09:07.757130 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: I0830 00:09:07.430622 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.250747 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.487642 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.312072 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.542742 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.371488 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-3]: I0830 00:09:07.598182 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.421554 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.429985 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.486861 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.653607 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.765600 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.542325 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.820999 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.708339 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.597192 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:09:07.875405 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.756797 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.931204 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:07.983837 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.250733 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.652321 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.765458 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.312170 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.707205 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:08.038516 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.820562 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: I0830 00:09:07.371448 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I0830 00:09:07.875232 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.756933 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: I0830 00:09:08.084518 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0830 00:09:07.421430 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.930369 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:08.084796 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0830 00:09:07.763939 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.819608 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:07.983428 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.874190 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.429551 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:workers restarted I0830 00:09:21.459562 281472882735808 gce_failure_handler_test.py:415] workers restarted [worker-3]: I0830 00:09:08.037691 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: I0830 00:09:07.486847 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:08.084175 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Training finished. [worker-1]: I0830 00:09:07.929225 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.542304 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:08.084469 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.597166 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:07.983243 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.652158 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:08.036854 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: I0830 00:09:07.707093 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:08.084421 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:Training finished. [worker-0]: I0830 00:09:07.756756 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0830 00:09:08.084623 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.763800 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.819564 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.874682 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.929116 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:07.983208 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:08.036281 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0830 00:09:08.084281 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0830 00:09:08.084489 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0830 00:09:21.479767 281473157069504 multi_process_runner.py:840] Subprocess with PID 1712715 (worker, 0) is now being started. [worker-0]: I0830 00:09:21.480374 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0830 00:09:21.489513 281473157069504 multi_process_runner.py:840] Subprocess with PID 1713040 (worker, 1) is now being started. [worker-1]: I0830 00:09:21.490061 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:09:21.495748 281473157069504 multi_process_runner.py:840] Subprocess with PID 1713129 (worker, 2) is now being started. [worker-3]: I0830 00:09:21.496564 281473157069504 multi_process_runner.py:840] Subprocess with PID 1713239 (worker, 3) is now being started. [worker-2]: I0830 00:09:21.496495 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:09:21.497060 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:42597", "localhost:44773", "localhost:33797", "localhost:35777"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-08-30 00:09:21.550034: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42597 [worker-0]: 2023-08-30 00:09:21.555216: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 7802817085549468713 [worker-3]: 2023-08-30 00:09:21.555451: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35777 [worker-0]: 2023-08-30 00:09:21.555456: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-08-30 00:09:21.573838: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33797 [worker-0]: 2023-08-30 00:09:21.574510: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 16872992077101245001 [worker-3]: 2023-08-30 00:09:21.574758: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2023-08-30 00:09:21.587562: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:44773 [worker-0]: 2023-08-30 00:09:21.596435: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 8593484920579802503 [worker-2]: 2023-08-30 00:09:21.596679: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:21.609888: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 3385089569009213709 [worker-1]: 2023-08-30 00:09:21.610822: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0830 00:09:21.615631 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:09:21.615260 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0830 00:09:21.614700 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:09:21.637739 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0830 00:09:21.669554 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-2]: I0830 00:09:21.669489 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:09:21.670170 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: I0830 00:09:21.670104 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:09:21.670346 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:09:21.670420 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: I0830 00:09:21.701174 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: I0830 00:09:21.701347 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0830 00:09:21.701983 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: I0830 00:09:21.701712 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:09:21.702229 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:09:21.701951 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:42597', 'localhost:44773', 'localhost:33797', 'localhost:35777']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0830 00:09:21.733612 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0830 00:09:21.733716 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0830 00:09:21.733746 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I0830 00:09:21.733402 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0830 00:09:21.734601 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: I0830 00:09:21.734724 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: I0830 00:09:21.734798 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-0]: I0830 00:09:21.734263 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Traceback (most recent call last): [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: Traceback (most recent call last): [worker-0]: Traceback (most recent call last): [worker-3]: Traceback (most recent call last): [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: I0830 00:09:21.735491 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0830 00:09:21.734783 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: I0830 00:09:21.735137 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: Instructions for updating: [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: I0830 00:09:21.735304 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: Instructions for updating: [worker-0]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: W0830 00:09:21.735637 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Instructions for updating: [worker-2]: Instructions for updating: [worker-0]: W0830 00:09:21.735417 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-3]: W0830 00:09:21.735851 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: INFO:tensorflow:Start training at 0 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Instructions for updating: [worker-2]: I0830 00:09:21.736012 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Start training at 0 [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: self.run() [worker-0]: I0830 00:09:21.735656 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start training at 0 [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: self.run() [worker-3]: I0830 00:09:21.736075 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-3]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-2]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: self._target(*self._args, **self._kwargs) [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: if self._termination_watcher_fn(): [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: if self._termination_watcher_fn(): [worker-1]: W0830 00:09:21.736290 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: Instructions for updating: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: I0830 00:09:21.736663 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:22.369308 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:22.381025 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:22.385111 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:22.378762 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:22.457172 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:22.458172 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:22.457276 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:22.457338 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:22.517292 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:22.517028 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:22.517271 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:22.517608 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:22.586781 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:22.586432 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:22.586307 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:22.586527 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:22.644813 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:22.645272 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:22.645074 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:22.645051 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f1de700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f1dfec0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:22.702895 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f1dfec0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:22.697380 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f1de700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f1de840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f1de520> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0830 00:09:22.713668 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f1de520> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0830 00:09:22.720409 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:22.740329 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:22.754631 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0830 00:09:22.718625 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f1de840> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:22.739979 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f1defc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df740> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:22.903717 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f1defc0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:22.903974 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df740> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0830 00:09:22.904077 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0830 00:09:22.904373 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df100> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:22.908875 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df100> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0830 00:09:22.909666 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:22.916457 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f1df1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0830 00:09:22.916852 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:22.925569 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:22.930389 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:22.938776 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:22.947139 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.014089 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.020878 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.023366 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.040738 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.101717 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.105150 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.104444 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.105179 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.170859 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.171578 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.170857 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.173829 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.230346 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.230375 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.230426 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.230814 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.288916 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.290000 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.289789 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.338285 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0830 00:09:23.687690 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.696955 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0830 00:09:23.697000 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0830 00:09:23.696142 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: I0830 00:09:23.705028 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.705117 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: I0830 00:09:23.705335 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.714009 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.810063 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.820270 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.824905 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.820450 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.922155 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.922405 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.925677 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.922066 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:23.980466 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:23.980807 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:23.980533 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:23.980643 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.047468 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.047497 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.055229 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.085949 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.150488 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.153941 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.154714 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.155107 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-0]: I0830 00:09:24.202596 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: I0830 00:09:24.202598 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: I0830 00:09:24.202804 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: I0830 00:09:24.202773 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.210854 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.211063 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.211090 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.231279 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.328397 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.328434 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.330980 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.334291 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.392006 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.392758 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.391672 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.410812 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.546289 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.561441 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.575998 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.547930 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.716573 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.715182 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.740778 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.752424 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.816843 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.825604 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.832017 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.841739 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0830 00:09:24.891625 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:09:24.891905 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0830 00:09:24.891960 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.900241 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.900717 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0830 00:09:24.902011 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.914224 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:24.916398 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:24.975272 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:24.976593 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:24.984767 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:25.000719 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:25.061643 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:25.061962 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:25.067173 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:25.062432 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Termination notice available. I0830 00:09:25.126354 281462385078752 gce_failure_handler_test.py:142] Termination notice available. INFO:tensorflow:Member single_worker has received termination notice. I0830 00:09:25.126701 281462385078752 failure_handling.py:701] Member single_worker has received termination notice. Exception ignored in: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 775, in __del__ self._stop_poll_termination_signal_thread() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 734, in _stop_poll_termination_signal_thread self._poll_termination_signal_thread.join() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1109, in join raise RuntimeError("cannot join current thread") RuntimeError: cannot join current thread [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:25.129528 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:25.136966 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:25.136547 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:25.160899 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:25.220158 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:25.220831 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:25.220834 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:25.221928 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:25.308238 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:25.320524 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:25.330508 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:25.363113 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0830 00:09:25.447126 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0830 00:09:25.447457 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0830 00:09:25.457645 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0830 00:09:25.457918 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0830 00:09:25.466558 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0830 00:09:25.466863 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0830 00:09:25.486626 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0830 00:09:25.486983 281473157069504 gce_failure_handler_test.py:244] Training finished. I0830 00:09:26.437589 281472882735808 multi_process_runner.py:646] worker-0 exit code: 0 I0830 00:09:26.437839 281472882735808 multi_process_runner.py:646] worker-1 exit code: 0 I0830 00:09:26.437983 281472882735808 multi_process_runner.py:646] worker-2 exit code: 0 I0830 00:09:26.438117 281472882735808 multi_process_runner.py:646] worker-3 exit code: 0 I0830 00:09:26.441477 281472882735808 multi_process_runner.py:662] Joining log reading threads. I0830 00:09:26.441828 281472882735808 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 22.2s I0830 00:09:26.690782 281472882735808 test_util.py:2477] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 22.2s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 44349 I0830 00:09:26.696705 281472882735808 test_util.py:3820] Using local port 44349 INFO:tensorflow:Using local port 44223 I0830 00:09:26.697212 281472882735808 test_util.py:3820] Using local port 44223 INFO:tensorflow:Using local port 38907 I0830 00:09:26.697573 281472882735808 test_util.py:3820] Using local port 38907 INFO:tensorflow:Using local port 34411 I0830 00:09:26.697917 281472882735808 test_util.py:3820] Using local port 34411 INFO:tensorflow:Cluster starting. I0830 00:09:26.829271 281472882735808 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0830 00:09:26.872071 281473157069504 multi_process_runner.py:840] Subprocess with PID 1728205 (worker, 0) is now being started. [worker-0]: I0830 00:09:26.872594 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0830 00:09:26.920281 281473157069504 multi_process_runner.py:840] Subprocess with PID 1728215 (worker, 1) is now being started. [worker-1]: I0830 00:09:26.920781 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:09:26.924728 281473157069504 multi_process_runner.py:840] Subprocess with PID 1728271 (worker, 2) is now being started. [worker-2]: I0830 00:09:26.925283 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:09:26.935962 281473157069504 multi_process_runner.py:840] Subprocess with PID 1728442 (worker, 3) is now being started. [worker-3]: I0830 00:09:26.936542 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-08-30 00:09:26.960277: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:44349 [worker-1]: 2023-08-30 00:09:26.964646: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:44223 [worker-0]: 2023-08-30 00:09:26.976461: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 8874041947359710771 [worker-0]: 2023-08-30 00:09:26.976699: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:26.978764: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 2392286516861189160 [worker-1]: 2023-08-30 00:09:26.978957: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-08-30 00:09:27.009465: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34411 [worker-2]: 2023-08-30 00:09:27.015799: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38907 [worker-0]: 2023-08-30 00:09:27.025620: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 6336928504826928183 [worker-3]: 2023-08-30 00:09:27.025894: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-08-30 00:09:27.025898: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:27.025694: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 4792443901974838244 [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:09:27.028229 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0830 00:09:27.028121 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:09:27.028138 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0830 00:09:27.028061 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0830 00:09:27.083668 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:09:27.084380 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-3]: I0830 00:09:27.084630 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:09:27.084962 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0830 00:09:27.085626 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:09:27.085876 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0830 00:09:27.106009 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: I0830 00:09:27.106706 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:09:27.106955 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:09:27.106448 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:09:27.107052 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:09:27.107298 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0830 00:09:27.166397 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0830 00:09:27.168500 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0830 00:09:27.169755 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-3]: Traceback (most recent call last): [worker-0]: I0830 00:09:27.170297 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: I0830 00:09:27.167003 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: if self._termination_watcher_fn(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0830 00:09:27.174047 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0830 00:09:27.170626 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: W0830 00:09:27.173187 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Traceback (most recent call last): [worker-2]: Traceback (most recent call last): [worker-1]: I0830 00:09:27.171610 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: INFO:tensorflow:Start training at 0 [worker-0]: self.run() [worker-2]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0830 00:09:27.198893 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0830 00:09:27.199374 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0830 00:09:27.199591 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: I0830 00:09:27.226378 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0830 00:09:27.267952 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0830 00:09:27.268459 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0830 00:09:27.268677 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0830 00:09:27.208469 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-3]: I0830 00:09:27.173409 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0830 00:09:27.208956 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0830 00:09:27.209177 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:27.406691 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:27.419164 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:27.424118 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:27.468976 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:27.628152 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:27.633915 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:27.635664 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:27.633657 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:27.738916 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:27.739747 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:27.751763 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:27.757971 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:27.890888 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:27.931610 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:27.940572 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:27.991277 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:28.255366 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:28.271183 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:28.270925 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:28.276546 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f420> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:28.406961 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f420> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:28.403007 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23f1a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f242b60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:28.410795 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f242b60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f243560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:28.416328 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f243560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:28.413972 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:28.431301 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:28.434013 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:28.461081 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xfffee7121080> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:28.532544 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xfffee7121080> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23fd80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243e20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:28.546040 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243e20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:28.539931 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23fd80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0830 00:09:28.540345 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:28.552844 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0830 00:09:28.553243 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:28.546494 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: I0830 00:09:28.533006 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:28.562445 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:28.544214 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:28.572697 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:28.590944 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:28.733374 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:28.733376 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:28.738367 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:28.757774 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:28.818808 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:28.818593 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:28.818850 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:28.819957 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:28.897888 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:28.897291 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:28.897987 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:28.897486 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:28.980108 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:28.983755 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:29.031194 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:29.101111 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:29.310661 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:29.323174 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:29.353599 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:29.776508 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0830 00:09:29.866300 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: I0830 00:09:29.853287 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: I0830 00:09:29.859730 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0830 00:09:29.896552 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:29.921545 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:29.891642 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.026546 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.029596 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.093872 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.099961 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.093902 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.120749 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.178974 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.178499 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.200656 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.192399 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.266078 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.284605 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.277044 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.284237 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.359780 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.366153 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.371609 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.407218 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.507605 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.508046 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.507891 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.507933 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0830 00:09:30.604545 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0830 00:09:30.604714 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: I0830 00:09:30.597530 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0830 00:09:30.606692 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.616184 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.629155 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.651044 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.629456 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.764672 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.770686 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.781516 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.817133 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:30.915935 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:30.942162 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:30.935608 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:30.989181 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:31.190878 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:31.200668 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:31.210686 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:31.224696 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:31.340613 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:31.346983 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:31.340675 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:31.370659 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:31.501149 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:31.508671 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:31.522871 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:31.590618 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0830 00:09:31.677861 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:09:31.707313 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: I0830 00:09:31.706470 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: I0830 00:09:31.701165 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:31.717689 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:31.727209 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:31.740592 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:31.740820 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:31.842889 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:31.852315 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:31.848968 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:31.861064 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:31.923682 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:31.924117 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:31.950363 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:31.970826 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:32.107239 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:32.115398 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:32.120716 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:32.161166 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:32.242014 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:32.242198 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:32.243366 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:32.255686 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:32.328504 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:32.356384 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:32.359626 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:32.370740 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0830 00:09:32.441607 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0830 00:09:32.441943 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0830 00:09:32.444173 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0830 00:09:32.444476 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0830 00:09:32.448717 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0830 00:09:32.449059 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0830 00:09:32.450424 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0830 00:09:32.450762 281473157069504 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0830 00:09:33.927486 281472882735808 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0830 00:09:33.959230 281472882735808 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0830 00:09:33.967210 281473157069504 multi_process_runner.py:840] Subprocess with PID 1741178 (worker, 0) is now being started. [worker-0]: I0830 00:09:33.967684 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0830 00:09:33.983507 281473157069504 multi_process_runner.py:840] Subprocess with PID 1741183 (worker, 1) is now being started. [worker-1]: I0830 00:09:33.983991 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:09:33.987213 281473157069504 multi_process_runner.py:840] Subprocess with PID 1741187 (worker, 2) is now being started. [worker-2]: I0830 00:09:33.987694 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:09:33.997767 281473157069504 multi_process_runner.py:840] Subprocess with PID 1741197 (worker, 3) is now being started. [worker-3]: I0830 00:09:33.998249 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:44349", "localhost:44223", "localhost:38907", "localhost:34411"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-08-30 00:09:34.003758: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:44349 [worker-2]: 2023-08-30 00:09:34.032167: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38907 [worker-3]: 2023-08-30 00:09:34.032140: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:34411 [worker-0]: 2023-08-30 00:09:34.066698: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 14009604942543006633 [worker-0]: 2023-08-30 00:09:34.066990: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-08-30 00:09:34.067443: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-08-30 00:09:34.067505: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:34.067244: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 218347275871667240 [worker-0]: 2023-08-30 00:09:34.067288: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 9501642576398555514 [worker-1]: 2023-08-30 00:09:34.079595: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:44223 [worker-0]: 2023-08-30 00:09:34.087209: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 5076338372835964291 [worker-1]: 2023-08-30 00:09:34.087439: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0830 00:09:34.089342 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0830 00:09:34.089359 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:09:34.090563 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:09:34.107604 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: I0830 00:09:34.144339 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: I0830 00:09:34.144513 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I0830 00:09:34.144545 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-0]: INFO:tensorflow:Check health not enabled. [worker-1]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:09:34.145026 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: I0830 00:09:34.145055 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: I0830 00:09:34.144926 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:09:34.145290 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:09:34.145262 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:09:34.145161 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0830 00:09:34.165036 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0830 00:09:34.165624 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:09:34.165860 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:44349', 'localhost:44223', 'localhost:38907', 'localhost:34411']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0830 00:09:34.198042 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0830 00:09:34.198487 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0830 00:09:34.201560 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0830 00:09:34.202813 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0830 00:09:34.203230 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0830 00:09:34.205811 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0830 00:09:34.205728 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-0]: I0830 00:09:34.205527 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: self.run() [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: I0830 00:09:34.206916 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-2]: I0830 00:09:34.207784 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: I0830 00:09:34.203466 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: Traceback (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: I0830 00:09:34.207411 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: self.run() [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: I0830 00:09:34.207686 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-3]: Instructions for updating: [worker-1]: self._target(*self._args, **self._kwargs) [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: W0830 00:09:34.207806 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-3]: Instructions for updating: [worker-1]: if self._termination_watcher_fn(): [worker-0]: W0830 00:09:34.208207 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: INFO:tensorflow:Start training at 0 [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: I0830 00:09:34.208047 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: if self._termination_watcher_fn(): [worker-0]: Instructions for updating: [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: INFO:tensorflow:Start training at 0 [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: I0830 00:09:34.209161 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: self._target(*self._args, **self._kwargs) [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: Traceback (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: self.run() [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0830 00:09:34.228608 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0830 00:09:34.229034 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0830 00:09:34.229265 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.318326 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.319711 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.324209 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.364072 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.442292 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.442414 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.448143 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.443189 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.507614 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.513184 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.507656 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.507637 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.572000 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.572357 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.572278 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.572254 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.629559 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.640634 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.629596 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.629597 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23d620> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:34.700461 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23d620> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23d800> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:34.702382 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23d800> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:34.700437 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.709189 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.710892 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f241940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:34.700781 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f241940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.709371 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.726968 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23e660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f242f20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:34.775949 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23e660> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:34.776281 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f242f20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-2]: W0830 00:09:34.776785 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f4c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0830 00:09:34.776314 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-1]: I0830 00:09:34.776621 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: I0830 00:09:34.777127 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:34.775855 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23f2e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0830 00:09:34.776224 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.784965 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.785073 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.784997 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.785374 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.842198 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.842379 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.842785 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.843239 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.899528 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.899641 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.899915 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.930553 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:34.987285 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:34.987534 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:34.991742 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:34.987541 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:35.060197 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:35.064270 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:35.064323 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:35.086328 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:35.152255 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:35.152398 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:35.152268 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:35.152855 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: I0830 00:09:35.347573 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: I0830 00:09:35.347835 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0830 00:09:35.347934 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0830 00:09:35.347773 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:35.356068 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:35.356096 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:35.359237 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:35.356636 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:35.518446 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:35.523977 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:35.531462 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:35.551413 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:35.640209 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:35.650162 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:35.654182 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:35.680116 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:35.770216 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:35.758644 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:35.786484 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:35.790170 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:35.899430 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:35.935000 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:35.935360 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:35.950738 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.095177 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.106541 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.112051 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.116493 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-0]: I0830 00:09:36.175764 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: I0830 00:09:36.176081 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0830 00:09:36.176140 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0830 00:09:36.175838 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.185112 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.191849 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.213709 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.241534 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.389555 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.400842 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.410885 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.411014 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.486376 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.486555 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.497691 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.497947 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.557783 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.557786 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.557912 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.557876 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.618817 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.619174 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.618818 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.619176 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.691035 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.688183 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.694586 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.704515 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0830 00:09:36.772316 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:09:36.772914 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0830 00:09:36.771621 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.781700 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.781137 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0830 00:09:36.786746 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.796109 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.797085 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:36.952610 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:36.959451 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:36.958459 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:36.990064 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:37.100973 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:37.101373 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:37.111482 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:37.135644 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:37.198562 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:37.198300 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:37.199355 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:37.207073 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:37.292222 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:37.297079 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:37.301857 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:37.335273 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:37.405983 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:37.414863 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:37.410498 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:37.430241 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0830 00:09:37.502982 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-1]: I0830 00:09:37.509464 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0830 00:09:37.503123 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0830 00:09:37.503334 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0830 00:09:37.509810 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0830 00:09:37.509372 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0830 00:09:37.503457 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0830 00:09:37.509720 281473157069504 gce_failure_handler_test.py:244] Training finished. I0830 00:09:37.947211 281472882735808 multi_process_runner.py:646] worker-0 exit code: 0 I0830 00:09:37.947450 281472882735808 multi_process_runner.py:646] worker-1 exit code: 0 I0830 00:09:37.947594 281472882735808 multi_process_runner.py:646] worker-2 exit code: 0 I0830 00:09:37.947728 281472882735808 multi_process_runner.py:646] worker-3 exit code: 0 I0830 00:09:37.950691 281472882735808 multi_process_runner.py:662] Joining log reading threads. I0830 00:09:37.950932 281472882735808 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 11.48s I0830 00:09:38.170695 281472882735808 test_util.py:2477] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 11.48s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 37771 I0830 00:09:38.172381 281472882735808 test_util.py:3820] Using local port 37771 INFO:tensorflow:Using local port 35265 I0830 00:09:38.172779 281472882735808 test_util.py:3820] Using local port 35265 INFO:tensorflow:Using local port 39327 I0830 00:09:38.173144 281472882735808 test_util.py:3820] Using local port 39327 INFO:tensorflow:Using local port 45349 I0830 00:09:38.173488 281472882735808 test_util.py:3820] Using local port 45349 INFO:tensorflow:Cluster starting. I0830 00:09:38.238010 281472882735808 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0830 00:09:38.283530 281473157069504 multi_process_runner.py:840] Subprocess with PID 1761014 (worker, 0) is now being started. [worker-0]: I0830 00:09:38.284029 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-08-30 00:09:38.324120: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37771 [worker-1]: I0830 00:09:38.351682 281473157069504 multi_process_runner.py:840] Subprocess with PID 1761046 (worker, 1) is now being started. [worker-0]: 2023-08-30 00:09:38.361656: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 925911546594822942 [worker-0]: 2023-08-30 00:09:38.361863: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: I0830 00:09:38.352182 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:09:38.378078 281473157069504 multi_process_runner.py:840] Subprocess with PID 1761051 (worker, 2) is now being started. [worker-2]: I0830 00:09:38.378557 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:09:38.386563 281473157069504 multi_process_runner.py:840] Subprocess with PID 1761139 (worker, 3) is now being started. [worker-1]: 2023-08-30 00:09:38.401558: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35265 [worker-3]: I0830 00:09:38.387048 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: 2023-08-30 00:09:38.423741: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:45349 [worker-0]: 2023-08-30 00:09:38.431413: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 5452841216453146005 [worker-2]: 2023-08-30 00:09:38.431598: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39327 [worker-1]: 2023-08-30 00:09:38.431654: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:38.437862: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 10513748903804772135 [worker-2]: 2023-08-30 00:09:38.438124: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:09:38.461839: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 9323308349084550971 [worker-3]: 2023-08-30 00:09:38.462210: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0830 00:09:38.464954 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:09:38.465065 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0830 00:09:38.466673 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:09:38.468317 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0830 00:09:38.522751 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:09:38.523299 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:09:38.523532 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0830 00:09:38.539783 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0830 00:09:38.540348 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:09:38.540585 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: I0830 00:09:38.544413 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:09:38.544419 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:09:38.544970 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: I0830 00:09:38.544952 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:09:38.545187 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:09:38.545204 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:37771', 'localhost:35265', 'localhost:39327', 'localhost:45349']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0830 00:09:38.626239 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0830 00:09:38.626245 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0830 00:09:38.628403 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: I0830 00:09:38.628403 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0830 00:09:38.647516 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0830 00:09:38.656435 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0830 00:09:38.656729 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-2]: Traceback (most recent call last): [worker-3]: I0830 00:09:38.657451 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0830 00:09:38.659715 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0830 00:09:38.660105 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0830 00:09:38.660325 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0830 00:09:38.668773 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0830 00:09:38.669327 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0830 00:09:38.669558 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0830 00:09:38.688769 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0830 00:09:38.689317 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0830 00:09:38.689549 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: Traceback (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0830 00:09:38.678826 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0830 00:09:38.679379 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0830 00:09:38.679620 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:38.774806 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:38.871101 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:38.905900 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:38.941591 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:39.073077 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:39.071832 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:39.095276 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:39.111449 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:39.213418 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:39.301138 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:39.323158 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:39.421653 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:39.557278 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:39.570367 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:39.590697 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:39.681022 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:39.754691 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:39.756720 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:39.770880 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:39.801148 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f243380> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f243060> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:40.086996 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f243380> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:40.087368 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f243060> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f241760> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:40.092923 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f241760> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:40.095650 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f243560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:40.096395 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f243560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:40.101453 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:40.104994 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:40.117445 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243e20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243f60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:09:40.232481 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243f60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:09:40.226728 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243e20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: I0830 00:09:40.232811 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: I0830 00:09:40.227075 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:09:40.239427 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243d80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f242700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:09:40.245251 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f242700> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: I0830 00:09:40.245585 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0830 00:09:40.239764 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:40.255733 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:40.260261 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:40.271852 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:40.284058 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:40.431505 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:40.437073 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:40.461069 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:40.521759 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:40.705418 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:40.720233 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:40.740431 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:40.791480 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:40.858891 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:40.861292 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:40.870980 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:40.900960 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:41.006381 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:41.020203 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:41.050657 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:41.040260 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:41.330585 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:41.339864 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:41.346934 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:41.350935 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0830 00:09:41.688081 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:41.696339 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0830 00:09:41.716590 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0830 00:09:41.709821 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: I0830 00:09:41.716271 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:41.730192 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:41.740851 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:41.740518 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:41.868517 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:41.893694 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:41.914273 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:41.920009 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:42.023422 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:42.060275 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:42.060274 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:42.080249 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:42.360933 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:42.370207 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:42.380747 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:42.400359 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:42.597008 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:42.610097 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:42.610162 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:42.620153 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:42.744918 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:42.748137 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:42.751208 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:42.750391 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0830 00:09:42.858281 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0830 00:09:42.864124 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0830 00:09:42.864882 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0830 00:09:42.870549 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:42.879367 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:42.890187 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:42.900801 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:42.910114 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.050350 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.070320 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.080185 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.080183 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.240329 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.240330 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.267045 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.260235 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.376158 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.380470 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.390191 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.390307 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.480388 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.480524 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.483538 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.511250 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.599737 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.608893 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.610164 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.624065 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-3]: I0830 00:09:43.685218 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: I0830 00:09:43.685542 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0830 00:09:43.685559 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.695026 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:09:43.699251 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.707773 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.723023 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.720289 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.812457 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.820108 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.814793 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.840294 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:43.910637 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.914295 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:43.911070 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:43.910905 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:44.002933 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:44.005276 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:43.996673 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:44.002084 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:44.076219 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:44.081061 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:44.100753 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:44.107059 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:09:44.196498 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:09:44.214609 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:09:44.196529 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:09:44.220454 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: I0830 00:09:44.272977 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0830 00:09:44.273289 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-0]: I0830 00:09:44.273285 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0830 00:09:44.274052 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0830 00:09:44.273566 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-1]: I0830 00:09:44.274324 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0830 00:09:44.275132 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0830 00:09:44.275413 281473157069504 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0830 00:09:46.336632 281472882735808 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0830 00:09:46.394718 281472882735808 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0830 00:09:46.496810 281473157069504 multi_process_runner.py:840] Subprocess with PID 1797592 (worker, 0) is now being started. [worker-3]: I0830 00:09:46.500789 281473157069504 multi_process_runner.py:840] Subprocess with PID 1797826 (worker, 3) is now being started. [worker-2]: I0830 00:09:46.500947 281473157069504 multi_process_runner.py:840] Subprocess with PID 1797731 (worker, 2) is now being started. [worker-1]: I0830 00:09:46.507498 281473157069504 multi_process_runner.py:840] Subprocess with PID 1797628 (worker, 1) is now being started. [worker-0]: I0830 00:09:46.497314 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: E0830 00:09:46.524438521 1797592 server_chttp2.cc:40] {"created":"@1693354186.524321939","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1693354186.524316263","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1693354186.524290795","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1693354186.524279829","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1693354186.524315293","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1693354186.524308862","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-08-30 00:09:46.524569: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-08-30 00:09:46.579259: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:780] Could not start gRPC server [worker-3]: I0830 00:09:46.501311 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:09:46.501391 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0830 00:09:46.507985 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:37771", "localhost:35265", "localhost:39327", "localhost:45349"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: 2023-08-30 00:09:46.606387: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:45349 [worker-0]: Process _Process-30: [worker-0]: Traceback (most recent call last): [worker-2]: 2023-08-30 00:09:46.645855: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39327 [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-0]: return self._actual_run() [worker-0]: ^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-0]: _run_main(main, args) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-0]: sys.exit(main(argv)) [worker-0]: ^^^^^^^^^^ [worker-1]: 2023-08-30 00:09:46.664419: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35265 [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: ^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-0]: six.reraise(*info.exc_info) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-0]: raise value [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-0]: return_value = fn(*args, **kwargs) [worker-0]: ^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-0]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ [worker-0]: CollectiveAllReduceExtended( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-0]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-0]: self._initialize_multi_worker(cluster_resolver) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-0]: context.context().ensure_initialized() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized [worker-0]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-0]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server E0830 00:13:56.406256 281472882735808 multi_process_runner.py:626] Timeout when joining for child processes. Terminating... I0830 00:13:56.406556 281472882735808 multi_process_runner.py:715] worker-0 has already exited. Not terminating. I0830 00:13:56.406865 281472882735808 multi_process_runner.py:721] worker-1 terminated with signal . I0830 00:13:56.407110 281472882735808 multi_process_runner.py:721] worker-2 terminated with signal . I0830 00:13:56.407332 281472882735808 multi_process_runner.py:721] worker-3 terminated with signal . [worker-3]: 2023-08-30 00:13:56.488815: W tensorflow/tsl/distributed_runtime/preemption/preemption_notifier.cc:89] SIGTERM caught at 2023-08-30T00:13:56.488737952+00:00 [worker-2]: 2023-08-30 00:13:56.546757: W tensorflow/tsl/distributed_runtime/preemption/preemption_notifier.cc:89] SIGTERM caught at 2023-08-30T00:13:56.546663066+00:00 [worker-1]: 2023-08-30 00:13:56.586332: W tensorflow/tsl/distributed_runtime/preemption/preemption_notifier.cc:89] SIGTERM caught at 2023-08-30T00:13:56.586237068+00:00 [worker-1]: 2023-08-30 00:13:56.736687: I tensorflow/core/common_runtime/eager/context_distributed_manager.cc:821] Preemption not exported to coordination service: UNAVAILABLE: failed to connect to all addresses [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/InsertKeyValue: [worker-1]: :{"created":"@1693354436.736520619","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3940,"referenced_errors":[{"created":"@1693354436.736517173","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":392,"grpc_status":14}]} [worker-3]: 2023-08-30 00:13:56.849724: I tensorflow/core/common_runtime/eager/context_distributed_manager.cc:821] Preemption not exported to coordination service: UNAVAILABLE: failed to connect to all addresses [worker-3]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/InsertKeyValue: [worker-3]: :{"created":"@1693354436.849568291","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3940,"referenced_errors":[{"created":"@1693354436.849563440","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":392,"grpc_status":14}]} [worker-2]: 2023-08-30 00:13:57.138110: I tensorflow/core/common_runtime/eager/context_distributed_manager.cc:821] Preemption not exported to coordination service: UNAVAILABLE: failed to connect to all addresses [worker-2]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/InsertKeyValue: [worker-2]: :{"created":"@1693354437.137803431","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3940,"referenced_errors":[{"created":"@1693354437.137800041","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":392,"grpc_status":14}]} E0830 00:14:26.407639 281472882735808 multi_process_runner.py:632] Timeout when waiting for child processes to print stacktrace. Sending SIGKILL... I0830 00:14:26.407934 281472882735808 multi_process_runner.py:715] worker-0 has already exited. Not terminating. I0830 00:14:26.408938 281472882735808 multi_process_runner.py:721] worker-1 terminated with signal . I0830 00:14:26.409816 281472882735808 multi_process_runner.py:721] worker-2 terminated with signal . I0830 00:14:26.410588 281472882735808 multi_process_runner.py:721] worker-3 terminated with signal . [ FAILED ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 300.23s I0830 00:14:38.403602 281472882735808 test_util.py:2477] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker): 300.23s [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 38371 I0830 00:14:38.411281 281472882735808 test_util.py:3820] Using local port 38371 INFO:tensorflow:Using local port 39025 I0830 00:14:38.411706 281472882735808 test_util.py:3820] Using local port 39025 INFO:tensorflow:Using local port 41091 I0830 00:14:38.412070 281472882735808 test_util.py:3820] Using local port 41091 INFO:tensorflow:Using local port 36381 I0830 00:14:38.412422 281472882735808 test_util.py:3820] Using local port 36381 INFO:tensorflow:Cluster starting. I0830 00:14:38.454571 281472882735808 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0830 00:14:38.557751 281473157069504 multi_process_runner.py:840] Subprocess with PID 2205102 (worker, 0) is now being started. [worker-0]: I0830 00:14:38.558297 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:14:38.590548 281473157069504 multi_process_runner.py:840] Subprocess with PID 2205389 (worker, 2) is now being started. [worker-3]: I0830 00:14:38.627562 281473157069504 multi_process_runner.py:840] Subprocess with PID 2205395 (worker, 3) is now being started. [worker-2]: I0830 00:14:38.591076 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:14:38.628092 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0830 00:14:38.654481 281473157069504 multi_process_runner.py:840] Subprocess with PID 2205161 (worker, 1) is now being started. [worker-1]: I0830 00:14:38.654956 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2023-08-30 00:14:38.665244: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38371 [worker-3]: 2023-08-30 00:14:38.673100: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:36381 [worker-0]: 2023-08-30 00:14:38.683371: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 1445334926987889651 [worker-0]: 2023-08-30 00:14:38.683473: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 9811908827555721549 [worker-0]: 2023-08-30 00:14:38.684061: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-08-30 00:14:38.674704: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:41091 [worker-2]: 2023-08-30 00:14:38.683743: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:14:38.687282: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 14749245169432288399 [worker-3]: 2023-08-30 00:14:38.687517: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-1]: 2023-08-30 00:14:38.688272: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39025 [worker-0]: 2023-08-30 00:14:38.693483: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 4746654052898378623 [worker-1]: 2023-08-30 00:14:38.693938: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:14:38.695951 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0830 00:14:38.696330 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: I0830 00:14:38.695991 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:14:38.696424 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I0830 00:14:38.751946 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: I0830 00:14:38.751939 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-3]: I0830 00:14:38.751964 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-2]: I0830 00:14:38.752110 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-1]: INFO:tensorflow:Check health not enabled. [worker-3]: INFO:tensorflow:Check health not enabled. [worker-2]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:14:38.752526 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: I0830 00:14:38.752560 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: I0830 00:14:38.752606 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: I0830 00:14:38.752731 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:14:38.752767 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:14:38.752796 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:14:38.752846 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:14:38.752972 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0830 00:14:38.785206 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0830 00:14:38.785422 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0830 00:14:38.785381 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0830 00:14:38.785533 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0830 00:14:38.786814 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: I0830 00:14:38.786393 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-0]: I0830 00:14:38.786108 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-3]: I0830 00:14:38.786419 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-0]: Traceback (most recent call last): [worker-2]: Traceback (most recent call last): [worker-1]: Traceback (most recent call last): [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: Traceback (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-0]: I0830 00:14:38.786753 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: I0830 00:14:38.787015 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-2]: I0830 00:14:38.787756 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: Instructions for updating: [worker-3]: I0830 00:14:38.786916 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0830 00:14:38.787708 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: W0830 00:14:38.787619 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-3]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0830 00:14:38.788275 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: W0830 00:14:38.787475 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0830 00:14:38.787950 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: INFO:tensorflow:Start training at 0 [worker-2]: Instructions for updating: [worker-3]: Instructions for updating: [worker-0]: self.run() [worker-1]: I0830 00:14:38.787858 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-1]: self.run() [worker-2]: INFO:tensorflow:Start training at 0 [worker-3]: INFO:tensorflow:Start training at 0 [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-2]: I0830 00:14:38.788516 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: I0830 00:14:38.787784 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: self.run() [worker-3]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-1]: if self._termination_watcher_fn(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: self._target(*self._args, **self._kwargs) [worker-2]: self._target(*self._args, **self._kwargs) [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: if self._termination_watcher_fn(): [worker-3]: if self._termination_watcher_fn(): [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:38.908099 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:38.908272 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:38.909195 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:38.910338 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:38.980257 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:restarting workers [worker-1]: I0830 00:14:38.980502 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0830 00:14:42.544331 281472882735808 gce_failure_handler_test.py:411] restarting workers [worker-3]: I0830 00:14:38.981339 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:38.980587 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.038675 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.038703 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.038705 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.038718 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.096755 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.097603 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.096359 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.096763 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.154442 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.153624 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ed40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: I0830 00:14:39.153175 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.153610 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ad40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:14:39.202888 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ed40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ed40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:14:39.202634 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ad40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0830 00:14:39.202895 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ed40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:14:39.202766 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.211679 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.211766 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ade0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: I0830 00:14:39.210928 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: W0830 00:14:39.259885 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0830 00:14:39.211682 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0830 00:14:39.259679 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ade0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ee80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:14:39.259934 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ede0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0830 00:14:39.260102 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0830 00:14:39.260221 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-0]: W0830 00:14:39.259769 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f23ee80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-3]: I0830 00:14:39.268865 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.260280 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0830 00:14:39.268805 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.260107 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.268236 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.323907 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.379338 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.433794 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.488515 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.542999 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0830 00:14:39.689645 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.697782 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.863571 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.917172 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:39.970978 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.024605 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.078410 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0830 00:14:40.125072 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.132719 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.186414 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.240300 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.294113 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.347914 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.401415 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0830 00:14:40.448213 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.455816 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.510262 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.563999 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.268792 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.617571 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.323914 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.671208 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.379271 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.725062 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.433788 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.771121 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0830 00:14:39.488518 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:14:40.771389 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0830 00:14:39.542927 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0830 00:14:39.689458 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.697251 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.863970 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.917183 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:39.970893 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.024568 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.078408 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0830 00:14:40.124959 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.132728 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.323912 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.186345 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.379279 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.323971 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.240281 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.433790 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.379465 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.294063 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.489573 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.433613 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.347914 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.542985 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.488322 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.401390 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.689345 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0830 00:14:39.543392 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: I0830 00:14:40.448097 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: I0830 00:14:39.697735 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.689731 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.455811 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.863413 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.697937 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.510273 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.917618 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.863534 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.563953 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:39.970872 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.916854 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.617515 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.024583 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:39.970705 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.671091 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.079516 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.024225 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.724775 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.124788 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: I0830 00:14:40.078032 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-0]: I0830 00:14:40.771000 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0830 00:14:40.132763 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Training finished. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:14:40.771270 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-3]: I0830 00:14:40.187016 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.125103 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.240306 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.132150 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.294078 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.347929 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.186073 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.239805 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.401420 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-2]: I0830 00:14:40.293591 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.447928 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.347441 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.455887 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.401478 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.510210 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.448260 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: I0830 00:14:40.563960 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.617529 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.455420 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.671083 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.509984 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.724797 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.563707 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.617248 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.770819 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Training finished. [worker-2]: I0830 00:14:40.670859 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:14:40.771136 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:14:40.724495 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0830 00:14:40.771161 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0830 00:14:40.771381 281473157069504 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:workers restarted I0830 00:15:03.690248 281472882735808 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0830 00:15:03.738650 281473157069504 multi_process_runner.py:840] Subprocess with PID 2210380 (worker, 0) is now being started. [worker-0]: I0830 00:15:03.739092 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0830 00:15:03.745993 281473157069504 multi_process_runner.py:840] Subprocess with PID 2210830 (worker, 2) is now being started. [worker-2]: I0830 00:15:03.746453 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0830 00:15:03.768457 281473157069504 multi_process_runner.py:840] Subprocess with PID 2210899 (worker, 3) is now being started. [worker-3]: I0830 00:15:03.768954 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-08-30 00:15:03.771927: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:38371 [worker-0]: 2023-08-30 00:15:03.780225: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 16065546397477009618 [worker-0]: 2023-08-30 00:15:03.780415: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-2]: 2023-08-30 00:15:03.797637: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:41091 [worker-3]: 2023-08-30 00:15:03.804605: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:36381 [worker-0]: 2023-08-30 00:15:03.807115: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 7431254632329658310 [worker-2]: 2023-08-30 00:15:03.807306: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-3]: 2023-08-30 00:15:03.807418: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: 2023-08-30 00:15:03.807176: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 16112085180206728369 [worker-1]: I0830 00:15:03.898600 281473157069504 multi_process_runner.py:840] Subprocess with PID 2210789 (worker, 1) is now being started. [worker-1]: I0830 00:15:03.899142 281473157069504 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:38371", "localhost:39025", "localhost:41091", "localhost:36381"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-1]: 2023-08-30 00:15:04.002846: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39025 [worker-0]: 2023-08-30 00:15:04.035261: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 12838329270693965121 [worker-1]: 2023-08-30 00:15:04.036687: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:303] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0830 00:15:04.038732 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0830 00:15:04.039243 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0830 00:15:04.047175 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0830 00:15:04.048668 281473157069504 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0830 00:15:04.098034 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0830 00:15:04.098713 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0830 00:15:04.098953 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: I0830 00:15:04.143292 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-2]: I0830 00:15:04.135560 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0830 00:15:04.143897 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0830 00:15:04.144140 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Check health not enabled. [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: I0830 00:15:04.136167 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0830 00:15:04.136407 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:15:04.148540 281473157069504 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0830 00:15:04.149189 281473157069504 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0830 00:15:04.149443 281473157069504 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:38371', 'localhost:39025', 'localhost:41091', 'localhost:36381']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0830 00:15:04.259253 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0830 00:15:04.260527 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0830 00:15:04.261165 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-3]: I0830 00:15:04.261235 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0830 00:15:04.270804 281473157069504 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0830 00:15:04.271976 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0830 00:15:04.272397 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Traceback (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-1]: I0830 00:15:04.272535 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0830 00:15:04.273123 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: Traceback (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: I0830 00:15:04.276906 281473157069504 failure_handling.py:683] Start polling for termination signal. [worker-0]: I0830 00:15:04.276493 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-1]: I0830 00:15:04.273434 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: Traceback (most recent call last): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-3]: I0830 00:15:04.277660 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-3]: Instructions for updating: [worker-0]: Instructions for updating: [worker-2]: Traceback (most recent call last): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 1038, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-1]: self.run() [worker-2]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: self._target(*self._args, **self._kwargs) [worker-2]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-1]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: W0830 00:15:04.279138 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: W0830 00:15:04.277238 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-0]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: I0830 00:15:04.279386 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: self.run() [worker-1]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: INFO:tensorflow:Start training at 0 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: I0830 00:15:04.277510 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: I0830 00:15:04.286233 281473157069504 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: if self._termination_watcher_fn(): [worker-2]: W0830 00:15:04.286700 281473157069504 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: Instructions for updating: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: INFO:tensorflow:Start training at 0 [worker-3]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-2]: I0830 00:15:04.286933 281473157069504 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.11/threading.py", line 975, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: ^^^^^^^^^^^^^^^^^^^^^^^^ [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:04.393685 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:04.394916 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:04.409587 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:04.439384 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:04.511293 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:04.511289 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:04.511628 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:04.511852 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:04.636284 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:04.640924 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:04.655476 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:04.657374 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:04.720089 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:04.720188 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:04.720418 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:04.720804 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:04.786449 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:04.786188 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:04.786578 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:04.787264 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f242200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:15:04.877023 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f242200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f241c60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:15:04.897379 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f241c60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f241a80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:15:04.903901 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f241a80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff8f242ac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:15:04.906454 281473157069504 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffff8f242ac0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:04.913089 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:04.910876 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:04.920775 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:04.940776 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f242b60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f242b60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0830 00:15:04.993452 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f242b60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0830 00:15:04.993657 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f242b60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0830 00:15:04.993801 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243920> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-3]: I0830 00:15:04.993898 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0830 00:15:04.994205 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0830 00:15:04.994104 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff8f243560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0830 00:15:04.993933 281473157069504 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffff8f243560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0830 00:15:04.994351 281473157069504 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.003244 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.003265 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.003556 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.004374 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.066174 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.066346 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.066363 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.066366 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.128293 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.128453 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.128285 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.128967 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.188907 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.188979 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.188919 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.189563 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.248669 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.248721 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.248779 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.249737 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.309845 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.309893 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.310111 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.310648 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I0830 00:15:05.515501 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: I0830 00:15:05.515697 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: I0830 00:15:05.515855 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0830 00:15:05.515950 281473157069504 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.524697 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.524690 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.524893 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.525458 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.585386 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.585503 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.585387 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.586046 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.644981 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.645359 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.645720 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.645828 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.704486 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.704525 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.704593 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.705013 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.764005 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.764021 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.764456 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.764702 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.824142 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.824224 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.824403 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.824685 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-3]: I0830 00:15:05.875367 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0830 00:15:05.875509 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: I0830 00:15:05.875624 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: I0830 00:15:05.875706 281473157069504 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.884443 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.884459 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.884662 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.885133 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:05.944338 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:05.944487 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:05.944860 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:05.944471 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.004803 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.004910 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.004873 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.006091 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.068275 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.068284 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.068413 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.069020 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.132260 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.132306 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.132385 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.133040 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.195102 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.195101 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.195262 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.195696 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: I0830 00:15:06.250813 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: I0830 00:15:06.251011 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: I0830 00:15:06.251192 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0830 00:15:06.251450 281473157069504 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.259773 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.260256 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.260435 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.260590 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.319979 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.319978 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.320116 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.320614 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.380788 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.381766 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.381870 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.381945 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.443196 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.443545 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.444114 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.451746 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.515259 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.515705 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.515253 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.515953 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0830 00:15:06.577228 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0830 00:15:06.577223 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0830 00:15:06.577458 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0830 00:15:06.577904 281473157069504 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-3]: I0830 00:15:06.630520 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: I0830 00:15:06.630840 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0830 00:15:06.630725 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: I0830 00:15:06.630896 281473157069504 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-3]: I0830 00:15:06.630888 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0830 00:15:06.631092 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0830 00:15:06.631230 281473157069504 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0830 00:15:06.631181 281473157069504 gce_failure_handler_test.py:244] Training finished. I0830 00:15:07.623506 281472882735808 multi_process_runner.py:646] worker-0 exit code: 0 I0830 00:15:07.623837 281472882735808 multi_process_runner.py:646] worker-1 exit code: 0 I0830 00:15:07.623991 281472882735808 multi_process_runner.py:646] worker-2 exit code: 0 I0830 00:15:07.624133 281472882735808 multi_process_runner.py:646] worker-3 exit code: 0 I0830 00:15:07.626966 281472882735808 multi_process_runner.py:662] Joining log reading threads. I0830 00:15:07.627225 281472882735808 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 29.36s I0830 00:15:07.769556 281472882735808 test_util.py:2477] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker): 29.36s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_manager_strategyoption_MWMSmultiworker ====================================================================== ERROR: test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker (__main__.GceFailureHandlingTest) GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_manager_strategyoption_MWMSmultiworker(api_wrapping_train=True, grace_period=0, input_arg='manager', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 417, in test_multiple_workers_preempted_consecutively mpr.join(timeout=250) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 637, in join self._reraise_if_subprocess_error(process_statuses) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 565, in _reraise_if_subprocess_error six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ CollectiveAllReduceExtended( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ self._initialize_strategy(self._cluster_resolver, devices=devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy self._initialize_multi_worker(cluster_resolver) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker context.context().ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ---------------------------------------------------------------------- Ran 7 tests in 383.580s FAILED (errors=1) ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 28.9s //tensorflow/c:c_api_function_test PASSED in 36.3s //tensorflow/c:c_api_test_cpu PASSED in 37.5s //tensorflow/c:c_test PASSED in 32.2s //tensorflow/c:env_test_cpu PASSED in 31.7s //tensorflow/c:kernels_test_cpu PASSED in 35.4s //tensorflow/c:ops_test PASSED in 24.0s //tensorflow/c:tf_status_helper_test PASSED in 0.1s //tensorflow/c:while_loop_test PASSED in 30.3s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 40.1s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 35.4s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 32.2s //tensorflow/c/eager:c_api_test_cpu PASSED in 42.9s //tensorflow/c/eager:custom_device_test PASSED in 36.8s //tensorflow/c/eager:dlpack_test_cpu PASSED in 31.2s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 37.0s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 29.6s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 57.1s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.1s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.2s //tensorflow/c/experimental/grappler:grappler_test PASSED in 33.7s //tensorflow/c/experimental/next_pluggable_device:tensor_pjrt_buffer_util_test PASSED in 6.6s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 0.7s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.6s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.6s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 22.4s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 14.8s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 17.5s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 14.6s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 16.5s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 18.1s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 19.0s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 34.2s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.1s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.5s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 0.5s //tensorflow/c/kernels:summary_op_test PASSED in 0.7s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.1s //tensorflow/cc:cc_op_gen_test PASSED in 0.8s //tensorflow/cc:client_client_session_test PASSED in 2.0s //tensorflow/cc:coordinator_test PASSED in 4.6s //tensorflow/cc:framework_cc_ops_test PASSED in 2.4s //tensorflow/cc:framework_gradient_checker_test PASSED in 3.0s //tensorflow/cc:framework_gradients_test PASSED in 4.6s //tensorflow/cc:framework_scope_test PASSED in 0.5s //tensorflow/cc:framework_while_gradients_test PASSED in 3.0s //tensorflow/cc:gradients_array_grad_test PASSED in 8.9s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 1.9s //tensorflow/cc:gradients_functional_grad_test PASSED in 3.3s //tensorflow/cc:gradients_image_grad_test PASSED in 7.0s //tensorflow/cc:gradients_linalg_grad_test PASSED in 2.3s //tensorflow/cc:gradients_manip_grad_test PASSED in 2.0s //tensorflow/cc:gradients_math_grad_test PASSED in 5.3s //tensorflow/cc:gradients_nn_grad_test PASSED in 4.8s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.4s //tensorflow/cc:ops_const_op_test PASSED in 0.5s //tensorflow/cc:ops_while_loop_test PASSED in 1.9s //tensorflow/cc:queue_runner_test PASSED in 12.4s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.4s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 37.8s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.2s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 33.0s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.5s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 36.7s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 31.6s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.5s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.4s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.1s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.1s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.0s //tensorflow/cc/saved_model:metrics_test PASSED in 0.1s //tensorflow/cc/saved_model:reader_test PASSED in 0.2s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 11.1s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 6.9s //tensorflow/cc/saved_model:util_test PASSED in 0.1s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 36.7s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 1.5s //tensorflow/compiler/aot:codegen_test PASSED in 34.0s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 22.7s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 11.1s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 5.7s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 7.8s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 27.8s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 7.0s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 21.4s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 28.3s //tensorflow/compiler/jit:device_util_test PASSED in 5.7s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 0.6s //tensorflow/compiler/jit:node_matchers_test PASSED in 0.7s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 11.3s //tensorflow/compiler/jit:shape_inference_test PASSED in 25.0s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 22.1s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 11.4s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 7.6s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 10.8s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 24.1s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 20.4s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 26.6s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 19.3s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 1.8s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.3s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 5.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 0.4s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 14.2s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 17.6s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 6.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:compose-uniform-quantized-type.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 15.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize_hlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:uniform-quantized-stablehlo-to-tfl.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:legalize-tensorlist.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 4.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 16.0s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 15.3s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants_offset.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op_offset.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 3.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:mix_tflite_stablehlo.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:stablehlo_custom_call.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_tensorlist_reserve.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 5.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 4.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test PASSED in 7.9s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_types_test PASSED in 28.9s //tensorflow/compiler/mlir/quantization/stablehlo:math_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/stablehlo:tf_type_utils_test PASSED in 19.6s //tensorflow/compiler/mlir/quantization/stablehlo:uniform_quantized_types_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 1.9s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibration_statistics_collector_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 23.9s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:constant_fold_test PASSED in 3.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 6.5s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/ops:tf_op_quant_spec_test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 87.1s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 24.9s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 10.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_xla_op_to_tf_op.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_hashtable_ops_as_args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_duplicate_resource_ops.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 15.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_weights.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 13.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_uniform_attribute_utils_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 39.9s //tensorflow/compiler/mlir/stablehlo:stablehlo_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 5.6s //tensorflow/compiler/mlir/tensorflow:call_graph_util_test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:data_dumper_logger_config_test PASSED in 5.4s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 17.1s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.1s //tensorflow/compiler/mlir/tensorflow:tf_mlir_translate_registration_test PASSED in 17.3s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:xla_rewrite_util_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:colocate_tpu_copy_with_dynamic_shape.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 3.9s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:embedding_program_key.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:embedding_sequencing.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 18.5s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 11.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 3.3s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 7.4s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 3.8s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 3.3s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_asset_sinking.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 10.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tfrt_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 3.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite_v2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_sharding_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow/tests:xla_validate_iputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:convert_mhlo_quant_to_int.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 18.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 36.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 37.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graph:convert_tensor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/transforms:verify_no_outside_compilation_markers_pass_test PASSED in 18.8s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_mlir_util_test PASSED in 5.0s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_tf_graph_test PASSED in 0.5s //tensorflow/compiler/mlir/tf2xla/api/v1:legalize_tf_test PASSED in 26.3s //tensorflow/compiler/mlir/tf2xla/internal:compilation_timer_test PASSED in 0.3s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_mlir_test PASSED in 22.2s //tensorflow/compiler/mlir/tf2xla/internal:legalize_tf_to_hlo_test PASSED in 21.0s //tensorflow/compiler/mlir/tf2xla/internal:mlir_pass_instrumentation_test PASSED in 8.3s //tensorflow/compiler/mlir/tf2xla/internal:test_matchers_test PASSED in 6.6s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-quant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 9.9s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tf2xla/transforms:legalization_op_config_test PASSED in 32.4s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 19.5s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 21.0s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 3.6s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 13.6s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 12.2s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 28.1s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 5.9s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 25.3s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 32.7s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 40.6s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 35.6s //tensorflow/compiler/mlir/tfrt/tests:batch_function_fallback_resource_variable_as_captured_tensor.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests:batch_function_lowering.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:convert_ref_variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:cross_device_transfer.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests:deduplicate_if_results.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:fuse_tpu_compile_and_execute_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops_mlrt.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:remove_device_attribute.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:sink_in_invariant_ops.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_fallback.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_lowering.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:xla_rewrite.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tfrt/tests/analysis:cost_analysis.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tfrt/tests/analysis:tensor_array_side_effect_analysis.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/analysis:update_op_cost_in_tfrt_mlir_test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/ir:fallback_opt.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/ir:tfrt_fallback_util_test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:assign_op_key.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/mlrt:async_while.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/mlrt:fuse_mlrt_ops.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tfrt/tests/mlrt:inline.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tfrt/tests/mlrt:parallelization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tf_to_mlrt.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tpu_conversions.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tfrt/tests/mlrt:while_to_map_fn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:basic.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate_failed.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:const_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:control_flow.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:decompose_resource_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:derived_attrs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:device_conversion.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:errors.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_canonicalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_inline.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes_multiple_callers.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_use_fallback_tensor.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:insert_fallback_tensor_copy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:merge_tf_if_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:optimize_tf_control_flow_side_effect.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:remove_tf_if_const_args.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:reorder_assert.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:side_effects.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline_refvar.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:whileop.mlir.test PASSED in 15.1s //tensorflow/compiler/mlir/tfrt/translate/mlrt:mlir_to_bytecode_test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:func_to_jit_invocations.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 17.4s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 2.2s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 15.4s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 15.6s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:adam_test_cpu PASSED in 15.4s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 10.1s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 16.9s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 17.9s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 27.3s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 11.4s //tensorflow/compiler/tests:case_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 9.9s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 14.1s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 19.3s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 15.2s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 35.6s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 11.7s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 12.6s //tensorflow/compiler/tests:cond_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 45.5s //tensorflow/compiler/tests:const_test_cpu PASSED in 13.8s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 19.9s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 16.6s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 13.1s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 10.0s //tensorflow/compiler/tests:eager_test_cpu PASSED in 57.6s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 16.0s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 11.3s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 10.0s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 11.6s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 18.4s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 19.9s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 26.4s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 13.0s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 13.6s //tensorflow/compiler/tests:function_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 11.2s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 12.0s //tensorflow/compiler/tests:gather_test_cpu PASSED in 53.0s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 78.1s //tensorflow/compiler/tests:jit_test_cpu PASSED in 44.0s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 18.0s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 13.8s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 10.8s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 36.9s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 13.9s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 14.8s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 43.3s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 39.8s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 20.8s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 22.3s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 12.0s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 15.9s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 15.9s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 12.0s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 12.3s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 11.3s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 47.1s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 12.9s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 11.8s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 13.6s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 14.9s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 17.8s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 36.3s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 16.8s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 13.9s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 25.1s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 60.6s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 20.7s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 28.7s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 25.4s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 31.5s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 19.8s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 19.2s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 34.5s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 22.2s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 11.1s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 9.9s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 13.6s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 15.5s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 17.2s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 18.5s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 15.0s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 16.9s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 38.9s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 20.7s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 12.3s //tensorflow/compiler/tests:while_test_cpu PASSED in 13.5s //tensorflow/compiler/tests:xla_call_module_no_platform_check_test_cpu PASSED in 13.4s //tensorflow/compiler/tests:xla_call_module_no_shape_assertions_check_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 15.3s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 11.1s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 13.1s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 15.6s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 17.9s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 39.3s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 56.8s //tensorflow/compiler/tests:xla_test_test PASSED in 9.7s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 7.7s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.7s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 1.3s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 1.4s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 24.8s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 6.6s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 0.5s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 6.3s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 2.3s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.9s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 10.4s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 20.5s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.7s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 17.9s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 17.9s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 6.1s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 2.0s //tensorflow/compiler/xla:array2d_test PASSED in 0.1s //tensorflow/compiler/xla:array3d_test PASSED in 0.1s //tensorflow/compiler/xla:array4d_test PASSED in 0.5s //tensorflow/compiler/xla:array_test PASSED in 0.1s //tensorflow/compiler/xla:bit_cast_test PASSED in 0.2s //tensorflow/compiler/xla:comparison_util_test PASSED in 0.2s //tensorflow/compiler/xla:debug_options_parsers_test PASSED in 0.1s //tensorflow/compiler/xla:index_util_test PASSED in 0.1s //tensorflow/compiler/xla:iterator_util_test PASSED in 0.1s //tensorflow/compiler/xla:layout_test PASSED in 0.1s //tensorflow/compiler/xla:layout_util_test PASSED in 0.3s //tensorflow/compiler/xla:literal_test PASSED in 0.6s //tensorflow/compiler/xla:parse_flags_from_env_test PASSED in 1.3s //tensorflow/compiler/xla:permutation_util_test PASSED in 5.1s //tensorflow/compiler/xla:primitive_util_test PASSED in 0.2s //tensorflow/compiler/xla:refcounting_hash_map_test PASSED in 0.1s //tensorflow/compiler/xla:reference_util_test PASSED in 0.2s //tensorflow/compiler/xla:shape_test PASSED in 0.2s //tensorflow/compiler/xla:shape_tree_test PASSED in 0.1s //tensorflow/compiler/xla:shape_util_test PASSED in 2.3s //tensorflow/compiler/xla:status_macros_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_reader_test PASSED in 1.1s //tensorflow/compiler/xla:text_literal_writer_test PASSED in 0.7s //tensorflow/compiler/xla:types_test PASSED in 0.1s //tensorflow/compiler/xla:util_test PASSED in 0.2s //tensorflow/compiler/xla:window_util_test PASSED in 0.6s //tensorflow/compiler/xla/client:padding_test PASSED in 0.9s //tensorflow/compiler/xla/client:xla_builder_test PASSED in 0.5s //tensorflow/compiler/xla/client/lib:arithmetic_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/client/lib:comparators_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/client/lib:constants_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/client/lib:logdet_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/client/lib:math_test_cpu PASSED in 13.9s //tensorflow/compiler/xla/client/lib:matrix_test_cpu PASSED in 11.0s //tensorflow/compiler/xla/client/lib:pooling_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/client/lib:qr_test_cpu PASSED in 13.8s //tensorflow/compiler/xla/client/lib:slicing_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/client/lib:sorting_test_cpu PASSED in 10.6s //tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test PASSED in 10.1s //tensorflow/compiler/xla/experiments/sm_bandwidth_benchmark:sm_bw_test PASSED in 0.1s //tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test PASSED in 16.0s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_solver_test PASSED in 1.2s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_test PASSED in 7.1s //tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test PASSED in 1.8s //tensorflow/compiler/xla/hlo/utils:hlo_live_range_test PASSED in 0.9s //tensorflow/compiler/xla/hlo/utils:hlo_matchers_test PASSED in 1.1s //tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:stream_assignment.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test PASSED in 0.1s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/utils:error_util_test PASSED in 0.4s //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/bufferization/hlo_one_shot_bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/analysis.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate_invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_simplification.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collect_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/batch_matmul.mlir.test PASSED in 4.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reshape_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d_map.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 17.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/optimize_linalg_ops.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/rewrite_forall_to_for.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 13.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test PASSED in 2.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-create-token-to-after-all.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-cross-replica-sum-to-all-reduce.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-dot-general-to-dot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-dot-to-dot-general.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-rng-to-linalg.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 3.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-torch-index-select-to-gather.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test PASSED in 14.9s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.1s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:host_callback_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:lru_cache_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_api_test PASSED in 1.6s //tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu PASSED in 10.1s //tensorflow/compiler/xla/pjrt:pjrt_compiler_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:pjrt_executable_test PASSED in 0.9s //tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test PASSED in 9.5s //tensorflow/compiler/xla/pjrt:semaphore_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:tf_pjrt_client_test PASSED in 9.1s //tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test PASSED in 9.6s //tensorflow/compiler/xla/pjrt:tracked_device_buffer_test PASSED in 6.7s //tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:transpose_test PASSED in 68.6s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test PASSED in 9.7s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test PASSED in 1.8s //tensorflow/compiler/xla/pjrt/distributed:topology_util_test PASSED in 0.1s //tensorflow/compiler/xla/python:outfeed_receiver_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/python:xplane_to_profile_instructions_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:array_test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:array_test_no_impl PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:client_test_no_impl PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:future_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_domain_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:memory_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:serdes_test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt:shape_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:sharding_serdes_test PASSED in 1.0s //tensorflow/compiler/xla/python/ifrt:sharding_test PASSED in 0.9s //tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt/ir/tests:executable_test_no_impl PASSED in 4.7s //tensorflow/compiler/xla/python/ifrt/ir/tests:ifrt_duplicated_callee_elimination.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_expansion.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_interface_verification.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/support:sharding_param_to_op_sharding_test PASSED in 0.3s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 14.8s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 8.7s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 7.8s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 6.9s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_executable_test_no_impl PASSED in 1.1s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_program_serdes_test PASSED in 2.1s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_serdes_test PASSED in 0.4s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_test PASSED in 7.6s //tensorflow/compiler/xla/python_api:xla_literal_test PASSED in 1.6s //tensorflow/compiler/xla/python_api:xla_shape_test PASSED in 1.7s //tensorflow/compiler/xla/rpc:grpc_client_test PASSED in 3.2s //tensorflow/compiler/xla/runtime:arguments_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:async_runtime_test PASSED in 0.8s //tensorflow/compiler/xla/runtime:custom_call_test PASSED in 2.0s //tensorflow/compiler/xla/runtime:diagnostics_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:executable_test PASSED in 3.1s //tensorflow/compiler/xla/runtime:ffi_test PASSED in 1.4s //tensorflow/compiler/xla/runtime:map_by_type_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:module_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:results_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:state_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:symbolic_shape_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:type_id_test PASSED in 0.2s //tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/service:algebraic_simplifier_test PASSED in 42.1s //tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test PASSED in 1.0s //tensorflow/compiler/xla/service:all_gather_combiner_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_gather_decomposer_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_combiner_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_contiguous_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_reduce_folder_test PASSED in 1.0s //tensorflow/compiler/xla/service:all_reduce_promotion_test PASSED in 1.6s //tensorflow/compiler/xla/service:all_reduce_reassociate_test PASSED in 1.6s //tensorflow/compiler/xla/service:all_reduce_simplifier_test PASSED in 0.7s //tensorflow/compiler/xla/service:ar_crs_combiner_test PASSED in 1.2s //tensorflow/compiler/xla/service:async_collective_creator_test PASSED in 1.2s //tensorflow/compiler/xla/service:async_op_canonicalizer_test PASSED in 0.7s //tensorflow/compiler/xla/service:batch_dot_simplification_test PASSED in 1.0s //tensorflow/compiler/xla/service:batchnorm_expander_test_cpu PASSED in 5.7s //tensorflow/compiler/xla/service:bfloat16_conversion_folding_test PASSED in 0.9s //tensorflow/compiler/xla/service:bfloat16_propagation_test PASSED in 1.1s //tensorflow/compiler/xla/service:bitcast_dtypes_expander_test PASSED in 1.5s //tensorflow/compiler/xla/service:broadcast_canonicalizer_test PASSED in 0.9s //tensorflow/compiler/xla/service:buffer_assignment_test PASSED in 6.9s //tensorflow/compiler/xla/service:call_graph_test PASSED in 0.7s //tensorflow/compiler/xla/service:call_inliner_test PASSED in 1.0s //tensorflow/compiler/xla/service:change_op_data_type_test PASSED in 0.8s //tensorflow/compiler/xla/service:collective_ops_utils_test PASSED in 0.2s //tensorflow/compiler/xla/service:collective_permute_decomposer_test PASSED in 1.1s //tensorflow/compiler/xla/service:collective_pipeliner_test PASSED in 3.8s //tensorflow/compiler/xla/service:collective_transformation_reorderer_test PASSED in 0.9s //tensorflow/compiler/xla/service:collectives_schedule_linearizer_test PASSED in 3.3s //tensorflow/compiler/xla/service:compilation_environments_test PASSED in 2.3s //tensorflow/compiler/xla/service:conditional_canonicalizer_test PASSED in 0.9s //tensorflow/compiler/xla/service:conditional_code_motion_test PASSED in 1.0s //tensorflow/compiler/xla/service:conditional_simplifier_test PASSED in 14.8s //tensorflow/compiler/xla/service:conditional_to_select_test PASSED in 1.2s //tensorflow/compiler/xla/service:constant_value_test PASSED in 0.2s //tensorflow/compiler/xla/service:convert_async_collectives_to_sync_test PASSED in 1.7s //tensorflow/compiler/xla/service:convert_mover_test PASSED in 0.8s //tensorflow/compiler/xla/service:convert_operand_folding_test PASSED in 0.9s //tensorflow/compiler/xla/service:convolution_4d_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:convolution_group_converter_test PASSED in 0.7s //tensorflow/compiler/xla/service:convolution_pred_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:copy_insertion_test PASSED in 1.6s //tensorflow/compiler/xla/service:custom_call_status_test PASSED in 0.1s //tensorflow/compiler/xla/service:defuser_test PASSED in 0.8s //tensorflow/compiler/xla/service:despecializer_test PASSED in 10.4s //tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test PASSED in 0.8s //tensorflow/compiler/xla/service:dot_decomposer_test PASSED in 1.5s //tensorflow/compiler/xla/service:dot_dimension_merger_test PASSED in 1.1s //tensorflow/compiler/xla/service:dot_merger_test PASSED in 0.8s //tensorflow/compiler/xla/service:dynamic_dimension_inference_test PASSED in 0.7s //tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test PASSED in 1.6s //tensorflow/compiler/xla/service:dynamic_index_splitter_test PASSED in 0.7s //tensorflow/compiler/xla/service:dynamic_padder_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/service:dynamic_parameter_binding_test PASSED in 0.6s //tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu PASSED in 17.7s //tensorflow/compiler/xla/service:flatten_call_graph_test PASSED in 0.9s //tensorflow/compiler/xla/service:float_normalization_test PASSED in 1.3s //tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test PASSED in 1.1s //tensorflow/compiler/xla/service:gather_expander_test PASSED in 2.1s //tensorflow/compiler/xla/service:gather_simplifier_test PASSED in 0.7s //tensorflow/compiler/xla/service:heap_simulator_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_alias_analysis_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_casting_utils_test PASSED in 7.2s //tensorflow/compiler/xla/service:hlo_computation_deduplicator_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_computation_test PASSED in 2.4s //tensorflow/compiler/xla/service:hlo_constant_folding_test PASSED in 5.4s //tensorflow/compiler/xla/service:hlo_cost_analysis_test PASSED in 6.7s //tensorflow/compiler/xla/service:hlo_creation_utils_test PASSED in 4.8s //tensorflow/compiler/xla/service:hlo_cse_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_dataflow_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_dce_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_domain_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_element_type_converter_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_execution_profile_test PASSED in 6.9s //tensorflow/compiler/xla/service:hlo_graph_dumper_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_input_output_alias_config_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_instruction_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_liveness_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_memory_scheduler_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_module_dce_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_module_metadata_test PASSED in 0.2s //tensorflow/compiler/xla/service:hlo_module_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_opcode_test PASSED in 0.1s //tensorflow/compiler/xla/service:hlo_ordering_test PASSED in 1.8s //tensorflow/compiler/xla/service:hlo_parser_test PASSED in 0.3s //tensorflow/compiler/xla/service:hlo_pass_pipeline_test PASSED in 0.6s //tensorflow/compiler/xla/service:hlo_phi_graph_test PASSED in 0.2s //tensorflow/compiler/xla/service:hlo_proto_util_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_reachability_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_rematerialization_test PASSED in 1.8s //tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_replication_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_schedule_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_sharding_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_value_semantics_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_verifier_test PASSED in 0.8s //tensorflow/compiler/xla/service:indexed_array_analysis_test PASSED in 1.5s //tensorflow/compiler/xla/service:instruction_fusion_test PASSED in 0.8s //tensorflow/compiler/xla/service:latency_hiding_scheduler_preparation_test PASSED in 0.8s //tensorflow/compiler/xla/service:latency_hiding_scheduler_test PASSED in 1.9s //tensorflow/compiler/xla/service:layout_assignment_test PASSED in 4.3s //tensorflow/compiler/xla/service:layout_normalization_test PASSED in 1.3s //tensorflow/compiler/xla/service:logistic_expander_test PASSED in 1.2s //tensorflow/compiler/xla/service:loop_schedule_linearizer_test PASSED in 1.7s //tensorflow/compiler/xla/service:map_inliner_test PASSED in 2.0s //tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test PASSED in 0.5s //tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.7s //tensorflow/compiler/xla/service:memory_space_assignment_test PASSED in 4.7s //tensorflow/compiler/xla/service:memory_space_propagation_test PASSED in 0.9s //tensorflow/compiler/xla/service:name_uniquer_test PASSED in 0.1s //tensorflow/compiler/xla/service:operand_upcaster_test PASSED in 0.8s //tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 0.8s //tensorflow/compiler/xla/service:pattern_matcher_gmock_test PASSED in 0.7s //tensorflow/compiler/xla/service:pattern_matcher_test PASSED in 2.6s //tensorflow/compiler/xla/service:profile_guided_latency_estimator_test PASSED in 1.1s //tensorflow/compiler/xla/service:real_imag_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:reduce_decomposer_test PASSED in 0.7s //tensorflow/compiler/xla/service:reduce_scatter_combiner_test PASSED in 0.8s //tensorflow/compiler/xla/service:reduce_scatter_decomposer_test PASSED in 0.8s //tensorflow/compiler/xla/service:reduce_scatter_reassociate_test PASSED in 1.7s //tensorflow/compiler/xla/service:reshape_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:reshape_mover_test PASSED in 1.5s //tensorflow/compiler/xla/service:result_caster_test PASSED in 0.8s //tensorflow/compiler/xla/service:root_instruction_sinker_test PASSED in 0.8s //tensorflow/compiler/xla/service:scatter_expander_test PASSED in 0.7s //tensorflow/compiler/xla/service:scatter_simplifier_test PASSED in 1.5s //tensorflow/compiler/xla/service:select_and_scatter_expander_test PASSED in 0.9s //tensorflow/compiler/xla/service:shape_inference_test PASSED in 0.2s //tensorflow/compiler/xla/service:shaped_buffer_test PASSED in 8.5s //tensorflow/compiler/xla/service:sharding_propagation_test PASSED in 9.1s //tensorflow/compiler/xla/service:sharding_remover_test PASSED in 1.7s //tensorflow/compiler/xla/service:simplify_fp_conversions_test PASSED in 1.5s //tensorflow/compiler/xla/service:slice_sinker_test PASSED in 0.7s //tensorflow/compiler/xla/service:sort_simplifier_test PASSED in 0.9s //tensorflow/compiler/xla/service:space_to_batch_converter_test PASSED in 2.2s //tensorflow/compiler/xla/service:stable_sort_expander_test PASSED in 1.5s //tensorflow/compiler/xla/service:stochastic_convert_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:stream_pool_test PASSED in 0.1s //tensorflow/compiler/xla/service:topk_rewriter_test PASSED in 4.7s //tensorflow/compiler/xla/service:transpose_folding_test PASSED in 1.6s //tensorflow/compiler/xla/service:tuple_points_to_analysis_test PASSED in 1.0s //tensorflow/compiler/xla/service:tuple_simplifier_test PASSED in 1.4s //tensorflow/compiler/xla/service:tuple_util_test PASSED in 0.8s //tensorflow/compiler/xla/service:value_range_test PASSED in 0.6s //tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_loop_analysis_test PASSED in 2.4s //tensorflow/compiler/xla/service:while_loop_concat_code_motion_test PASSED in 0.8s //tensorflow/compiler/xla/service:while_loop_constant_sinking_test PASSED in 1.0s //tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.4s //tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test PASSED in 0.7s //tensorflow/compiler/xla/service:while_loop_simplifier_test PASSED in 1.3s //tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test PASSED in 1.0s //tensorflow/compiler/xla/service:while_util_test PASSED in 1.7s //tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test PASSED in 8.3s //tensorflow/compiler/xla/service:xla_debug_info_manager_test PASSED in 0.9s //tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test PASSED in 0.8s //tensorflow/compiler/xla/service/cpu:conv_canonicalization_test PASSED in 1.6s //tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test PASSED in 1.1s //tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test PASSED in 2.9s //tensorflow/compiler/xla/service/cpu:ir_emission_utils_test PASSED in 2.8s //tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test PASSED in 3.4s //tensorflow/compiler/xla/service/cpu:runtime_fft_test PASSED in 0.4s //tensorflow/compiler/xla/service/cpu:shape_partition_test PASSED in 0.8s //tensorflow/compiler/xla/service/cpu:xfeed_manager_test PASSED in 0.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test PASSED in 0.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test PASSED in 9.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 7.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test PASSED in 35.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test PASSED in 7.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test PASSED in 8.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test PASSED in 9.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test PASSED in 11.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test PASSED in 7.9s //tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test PASSED in 9.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test PASSED in 8.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test PASSED in 9.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test PASSED in 6.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test PASSED in 9.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test PASSED in 8.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_while_test PASSED in 9.0s //tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 8.8s //tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:autotuner_util_test PASSED in 0.1s //tensorflow/compiler/xla/service/gpu:backend_configs_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:copy_fusion_test PASSED in 2.9s //tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test PASSED in 12.0s //tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test PASSED in 3.4s //tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:fusion_wrapper_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test PASSED in 2.3s //tensorflow/compiler/xla/service/gpu:gpu_async_collective_annotator_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:gpu_convert_async_collectives_to_sync_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:gpu_cost_model_stats_collection_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:gpu_fusible_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:gpu_performance_model_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test PASSED in 0.2s //tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:hlo_traversal_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:instruction_fusion_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:ir_emission_utils_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:matmul_utils_test PASSED in 2.6s //tensorflow/compiler/xla/service/gpu:move_copy_to_users_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:multi_output_fusion_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.1s //tensorflow/compiler/xla/service/gpu:priority_fusion_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:reduction_splitter_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:softmax_rewriter_triton_test PASSED in 2.2s //tensorflow/compiler/xla/service/gpu:target_util_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:topk_splitter_test PASSED in 34.9s //tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:while_transformer_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test PASSED in 0.3s //tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 2.2s //tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 1.6s //tensorflow/compiler/xla/service/graphcycles:graphcycles_test PASSED in 2.5s //tensorflow/compiler/xla/service/graphcycles:ordered_set_test PASSED in 0.2s //tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test PASSED in 9.3s //tensorflow/compiler/xla/service/llvm_ir:ir_array_test PASSED in 1.2s //tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 0.8s //tensorflow/compiler/xla/service/spmd:collective_permute_motion_test PASSED in 0.9s //tensorflow/compiler/xla/service/spmd:partition_assignment_test PASSED in 0.7s //tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 0.9s //tensorflow/compiler/xla/service/spmd:spmd_partitioner_test PASSED in 4.7s //tensorflow/compiler/xla/service/spmd:spmd_prepare_test PASSED in 1.0s //tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 1.0s //tensorflow/compiler/xla/service/spmd:whole_graph_manual_pass_test PASSED in 0.9s //tensorflow/compiler/xla/stream_executor:dnn_test PASSED in 0.2s //tensorflow/compiler/xla/stream_executor:stream_test PASSED in 0.2s //tensorflow/compiler/xla/stream_executor/host:host_stream_test PASSED in 0.2s //tensorflow/compiler/xla/stream_executor/tpu:c_api_conversions_test PASSED in 0.4s //tensorflow/compiler/xla/tests:all_reduce_test_cpu PASSED in 9.3s //tensorflow/compiler/xla/tests:axpy_simple_test_cpu PASSED in 10.0s //tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:binop_scaling_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests:bitcast_convert_test_cpu PASSED in 9.5s //tensorflow/compiler/xla/tests:broadcast_simple_test_cpu PASSED in 9.9s //tensorflow/compiler/xla/tests:broadcast_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests:buffer_donation_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests:call_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests:check_execution_arity_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:cholesky_test_cpu PASSED in 16.1s //tensorflow/compiler/xla/tests:client_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:collective_ops_test_cpu PASSED in 14.3s //tensorflow/compiler/xla/tests:collective_pipeliner_execution_test_cpu PASSED in 12.4s //tensorflow/compiler/xla/tests:compilation_cache_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests:compute_constant_test_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests:concat_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests:constants_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests:convert_test_cpu PASSED in 14.7s //tensorflow/compiler/xla/tests:copy_test_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests:custom_call_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:deallocation_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu PASSED in 9.8s //tensorflow/compiler/xla/tests:deep_graph_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:fft_test_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests:float8_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:floor_ceil_test_cpu PASSED in 10.0s //tensorflow/compiler/xla/tests:fmax_fmin_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:gather_operation_test_cpu PASSED in 12.6s //tensorflow/compiler/xla/tests:get_dimension_size_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:half_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:hlo_metadata_test PASSED in 8.3s //tensorflow/compiler/xla/tests:literal_test_util_test PASSED in 5.0s //tensorflow/compiler/xla/tests:local_client_allocation_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:local_client_aot_test PASSED in 0.1s //tensorflow/compiler/xla/tests:log_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:map_test_cpu PASSED in 10.2s //tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu PASSED in 26.3s //tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:multiple_devices_on_host_test PASSED in 8.3s //tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu PASSED in 9.8s //tensorflow/compiler/xla/tests:onednn_matmul_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:pad_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:pred_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:reduce_hlo_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests:reduce_precision_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:replay_test_cpu PASSED in 9.8s //tensorflow/compiler/xla/tests:reshape_motion_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests:reverse_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests:sample_text_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:scatter_test_cpu PASSED in 12.7s //tensorflow/compiler/xla/tests:select_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests:test_utils_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:tile_assignment_test PASSED in 0.1s //tensorflow/compiler/xla/tests:token_hlo_test_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests:topk_test_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests:transfer_manager_test_cpu PASSED in 16.0s //tensorflow/compiler/xla/tests:transpose_test_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests:tuple_test_cpu PASSED in 9.7s //tensorflow/compiler/xla/tests:unary_op_test_cpu PASSED in 9.8s //tensorflow/compiler/xla/tests:value_inference_test_cpu PASSED in 16.7s //tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu PASSED in 9.5s //tensorflow/compiler/xla/tests:while_test_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_000000_cpu PASSED in 9.3s //tensorflow/compiler/xla/tests/fuzz:rand_000003_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_000005_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000006_cpu PASSED in 9.4s //tensorflow/compiler/xla/tests/fuzz:rand_000007_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests/fuzz:rand_000008_cpu PASSED in 9.7s //tensorflow/compiler/xla/tests/fuzz:rand_000009_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000013_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_000015_cpu PASSED in 9.7s //tensorflow/compiler/xla/tests/fuzz:rand_000016_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests/fuzz:rand_000017_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_000018_cpu PASSED in 9.5s //tensorflow/compiler/xla/tests/fuzz:rand_000019_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests/fuzz:rand_000020_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_000022_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests/fuzz:rand_000024_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000025_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests/fuzz:rand_000026_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000030_cpu PASSED in 9.4s //tensorflow/compiler/xla/tests/fuzz:rand_000031_cpu PASSED in 6.3s //tensorflow/compiler/xla/tests/fuzz:rand_000032_cpu PASSED in 10.1s //tensorflow/compiler/xla/tests/fuzz:rand_000033_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests/fuzz:rand_000034_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests/fuzz:rand_000035_cpu PASSED in 9.6s //tensorflow/compiler/xla/tests/fuzz:rand_000036_cpu PASSED in 13.3s //tensorflow/compiler/xla/tests/fuzz:rand_000039_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000040_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests/fuzz:rand_000041_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests/fuzz:rand_000043_cpu PASSED in 12.2s //tensorflow/compiler/xla/tests/fuzz:rand_000049_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000053_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests/fuzz:rand_000056_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests/fuzz:rand_000059_cpu PASSED in 25.5s //tensorflow/compiler/xla/tests/fuzz:rand_000061_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests/fuzz:rand_000062_cpu PASSED in 13.3s //tensorflow/compiler/xla/tests/fuzz:rand_000064_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_000066_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests/fuzz:rand_000069_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_000071_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_000077_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests/fuzz:rand_000078_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000079_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests/fuzz:rand_000081_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests/fuzz:rand_000084_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_000085_cpu PASSED in 6.0s //tensorflow/compiler/xla/tests/fuzz:rand_000086_cpu PASSED in 9.1s //tensorflow/compiler/xla/tests/fuzz:rand_000088_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000089_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests/fuzz:rand_000090_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_000092_cpu PASSED in 11.0s //tensorflow/compiler/xla/tests/fuzz:rand_000094_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests/fuzz:rand_000095_cpu PASSED in 6.9s //tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test PASSED in 1.4s //tensorflow/compiler/xla/tools:hlo_extractor_test PASSED in 1.5s //tensorflow/compiler/xla/tools:hlo_module_loader_test PASSED in 0.8s //tensorflow/compiler/xla/tools:hlo_slicer_test PASSED in 0.7s //tensorflow/compiler/xla/tools:interactive_graphviz_bin_test PASSED in 0.6s //tensorflow/compiler/xla/tools:run_hlo_module_bin_test PASSED in 0.5s //tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test PASSED in 1.3s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test PASSED in 1.3s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test PASSED in 1.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:send_recv.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:stacktrace_to_location.hlo.test PASSED in 0.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test PASSED in 1.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test PASSED in 2.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test PASSED in 2.1s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_stacktrace.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test PASSED in 2.1s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test PASSED in 1.8s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test PASSED in 2.7s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test PASSED in 1.2s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 13.1s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 40.0s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 1.5s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.4s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.2s //tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests PASSED in 2.2s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.6s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.1s //tensorflow/core/api_def:api_test PASSED in 3.0s //tensorflow/core/api_def:update_api_def_test PASSED in 0.2s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 0.6s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.5s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 1.1s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 1.2s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 4.8s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 1.0s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.6s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 2.5s //tensorflow/core/common_runtime:cost_util_test PASSED in 1.4s //tensorflow/core/common_runtime:device_mgr_test PASSED in 1.1s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.6s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 0.8s //tensorflow/core/common_runtime:device_set_test PASSED in 1.2s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 2.1s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 3.6s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.0s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 1.0s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.7s //tensorflow/core/common_runtime:executor_test PASSED in 1.7s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 2.2s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 1.2s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 0.9s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.9s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 2.7s //tensorflow/core/common_runtime:graph_runner_test PASSED in 1.3s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 5.8s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 0.7s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.9s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 1.1s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 0.9s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 2.0s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 1.9s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 2.2s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 6.7s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.5s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.5s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.1s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.3s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.6s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 1.2s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 7.1s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:pending_counts_test PASSED in 0.9s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 3.4s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 1.6s //tensorflow/core/common_runtime:placer_test PASSED in 1.0s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 1.7s //tensorflow/core/common_runtime:process_util_test PASSED in 0.4s //tensorflow/core/common_runtime:quantize_training_test PASSED in 2.9s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.1s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.6s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.5s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.1s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 2.5s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 6.2s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 4.1s //tensorflow/core/common_runtime:session_test PASSED in 1.7s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 1.1s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 1.0s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 1.8s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.0s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 29.8s //tensorflow/core/common_runtime/eager:context_test PASSED in 13.8s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 12.3s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 12.7s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 1.0s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 14.2s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 10.4s //tensorflow/core/common_runtime/eager:execute_test PASSED in 27.7s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 0.9s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 16.4s //tensorflow/core/common_runtime/eager:placement_test PASSED in 11.1s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 10.8s //tensorflow/core/common_runtime/eager:summary_optimizer_test PASSED in 0.3s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 10.2s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 11.7s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.3s //tensorflow/core/common_runtime/gpu:gpu_serving_device_selector_test PASSED in 0.1s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 33.3s //tensorflow/core/common_runtime/next_pluggable_device/c:tf_rendezvous_c_api_conversions_test PASSED in 0.2s //tensorflow/core/config:flags_py_test PASSED in 10.9s //tensorflow/core/config:flags_test PASSED in 0.2s //tensorflow/core/data:compression_utils_test PASSED in 2.0s //tensorflow/core/data:dataset_utils_test PASSED in 1.0s //tensorflow/core/data:hash_utils_test PASSED in 1.2s //tensorflow/core/data:metric_utils_test PASSED in 5.8s //tensorflow/core/data:name_utils_test PASSED in 0.2s //tensorflow/core/data:rewrite_utils_test PASSED in 0.9s //tensorflow/core/data:serialization_utils_test PASSED in 0.8s //tensorflow/core/data:snapshot_utils_test PASSED in 0.6s //tensorflow/core/data:split_utils_test PASSED in 0.8s //tensorflow/core/data:standalone_save_restore_test PASSED in 1.9s //tensorflow/core/data:standalone_test PASSED in 4.1s //tensorflow/core/data:tfdataz_metrics_test PASSED in 1.7s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.8s //tensorflow/core/data/service:auto_scaler_test PASSED in 0.3s //tensorflow/core/data/service:common_test PASSED in 0.4s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.7s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 2.1s //tensorflow/core/data/service:data_service_test PASSED in 14.3s //tensorflow/core/data/service:data_transfer_test PASSED in 0.7s //tensorflow/core/data/service:dataset_store_test PASSED in 0.6s //tensorflow/core/data/service:dispatcher_client_test PASSED in 5.0s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.6s //tensorflow/core/data/service:graph_rewriters_test PASSED in 1.4s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 2.5s //tensorflow/core/data/service:grpc_util_test PASSED in 0.7s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 3.0s //tensorflow/core/data/service:journal_test PASSED in 0.5s //tensorflow/core/data/service:logging_utils_test PASSED in 0.2s //tensorflow/core/data/service:task_runner_test PASSED in 3.3s //tensorflow/core/data/service:test_util_test PASSED in 1.7s //tensorflow/core/data/service:url_test PASSED in 0.5s //tensorflow/core/data/service:utils_test PASSED in 1.0s //tensorflow/core/data/service:validate_utils_test PASSED in 0.2s //tensorflow/core/data/service:worker_client_test PASSED in 2.9s //tensorflow/core/data/service:worker_impl_test PASSED in 2.6s //tensorflow/core/data/service/client:data_service_client_test PASSED in 4.0s //tensorflow/core/data/service/client:utils_test PASSED in 2.8s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.3s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 20.4s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.7s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.1s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 3.5s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 1.1s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 4.1s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 2.0s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.1s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.8s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.1s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 3.6s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 1.2s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.5s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 0.8s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.1s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.7s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.2s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.1s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.6s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.1s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 1.8s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.4s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.5s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 26.7s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 16.2s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 36.4s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 48.1s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 28.2s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 4.0s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 0.9s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 1.0s //tensorflow/core/example:example_parser_configuration_test PASSED in 1.2s //tensorflow/core/example:feature_util_test PASSED in 0.1s //tensorflow/core/framework:allocator_test PASSED in 3.8s //tensorflow/core/framework:attr_value_util_test PASSED in 0.9s //tensorflow/core/framework:batch_util_test PASSED in 1.6s //tensorflow/core/framework:bfloat16_test PASSED in 0.9s //tensorflow/core/framework:common_shape_fns_test PASSED in 1.1s //tensorflow/core/framework:dataset_test PASSED in 0.7s //tensorflow/core/framework:device_base_test PASSED in 1.1s //tensorflow/core/framework:disable_jit_test PASSED in 0.8s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.5s //tensorflow/core/framework:framework_op_segment_test PASSED in 1.0s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.2s //tensorflow/core/framework:framework_run_handler_test PASSED in 2.0s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 1.9s //tensorflow/core/framework:full_type_inference_util_test PASSED in 11.4s //tensorflow/core/framework:full_type_util_test PASSED in 1.8s //tensorflow/core/framework:function_test PASSED in 1.1s //tensorflow/core/framework:graph_def_util_test PASSED in 1.1s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 1.1s //tensorflow/core/framework:kernel_def_builder_test PASSED in 0.8s //tensorflow/core/framework:kernel_def_util_test PASSED in 0.8s //tensorflow/core/framework:memory_types_test PASSED in 1.3s //tensorflow/core/framework:model_test PASSED in 1.2s //tensorflow/core/framework:node_def_builder_test PASSED in 1.3s //tensorflow/core/framework:node_def_util_test PASSED in 0.8s //tensorflow/core/framework:node_properties_test PASSED in 1.3s //tensorflow/core/framework:op_compatibility_test PASSED in 0.7s //tensorflow/core/framework:op_def_builder_test PASSED in 0.8s //tensorflow/core/framework:op_def_util_test PASSED in 2.0s //tensorflow/core/framework:op_kernel_test PASSED in 0.9s //tensorflow/core/framework:op_registration_test PASSED in 1.5s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 1.3s //tensorflow/core/framework:rendezvous_test PASSED in 3.5s //tensorflow/core/framework:resource_handle_test PASSED in 0.2s //tensorflow/core/framework:resource_mgr_test PASSED in 2.4s //tensorflow/core/framework:resource_op_kernel_test PASSED in 1.7s //tensorflow/core/framework:shape_inference_test PASSED in 11.4s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 1.1s //tensorflow/core/framework:tensor_matcher_test PASSED in 0.8s //tensorflow/core/framework:tensor_shape_test PASSED in 7.2s //tensorflow/core/framework:tensor_slice_test PASSED in 1.2s //tensorflow/core/framework:tensor_test PASSED in 44.4s //tensorflow/core/framework:tensor_testutil_test PASSED in 2.0s //tensorflow/core/framework:tensor_util_test PASSED in 1.0s //tensorflow/core/framework:tracking_allocator_test PASSED in 17.0s //tensorflow/core/framework:types_test PASSED in 1.0s //tensorflow/core/framework:variant_op_registry_test PASSED in 19.1s //tensorflow/core/framework:variant_test PASSED in 1.2s //tensorflow/core/framework/registration:registration_test PASSED in 0.5s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 14.6s //tensorflow/core/function/capture:capture_container_test PASSED in 10.3s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 24.2s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 24.2s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 10.6s //tensorflow/core/function/polymorphism:function_type_test PASSED in 9.4s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 10.0s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 41.9s //tensorflow/core/function/trace_type:custom_nest_trace_type_test PASSED in 10.0s //tensorflow/core/function/trace_type:default_types_test PASSED in 10.4s //tensorflow/core/function/trace_type:serialization_test PASSED in 12.6s //tensorflow/core/function/trace_type:trace_type_test PASSED in 17.0s //tensorflow/core/graph:algorithm_test PASSED in 0.9s //tensorflow/core/graph:collective_order_test PASSED in 1.9s //tensorflow/core/graph:control_flow_test PASSED in 1.5s //tensorflow/core/graph:costmodel_test PASSED in 1.5s //tensorflow/core/graph:edgeset_test PASSED in 1.1s //tensorflow/core/graph:graph_debug_info_builder_test PASSED in 0.9s //tensorflow/core/graph:graph_def_builder_test PASSED in 1.0s //tensorflow/core/graph:graph_partition_test PASSED in 0.9s //tensorflow/core/graph:graph_test PASSED in 1.1s //tensorflow/core/graph:node_builder_test PASSED in 0.8s //tensorflow/core/graph:optimizer_cse_test PASSED in 1.1s //tensorflow/core/graph:subgraph_test PASSED in 1.2s //tensorflow/core/graph:tensor_id_test PASSED in 1.0s //tensorflow/core/graph:validate_test PASSED in 1.1s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.4s //tensorflow/core/graph/regularization:util_test PASSED in 0.2s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.5s //tensorflow/core/grappler:graph_view_test PASSED in 1.7s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 1.6s //tensorflow/core/grappler:grappler_item_test PASSED in 1.4s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 1.3s //tensorflow/core/grappler:utils_test PASSED in 2.5s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 24.4s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 2.1s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 1.8s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.1s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.2s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 3.4s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.4s //tensorflow/core/grappler/costs:utils_test PASSED in 1.8s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.4s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.7s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 2.8s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 1.9s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 1.2s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 2.1s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 2.2s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 2.8s //tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu PASSED in 2.6s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 4.5s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 2.5s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.9s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 1.9s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.3s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 1.8s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 1.9s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 8.6s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 1.4s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 2.6s //tensorflow/core/grappler/optimizers:remapper_test_cpu PASSED in 3.0s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 2.1s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:inject_io_prefetch_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:remove_compression_map_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 1.2s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 1.0s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.5s //tensorflow/core/grappler/utils:frame_test PASSED in 0.1s //tensorflow/core/grappler/utils:functions_test PASSED in 1.4s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.4s //tensorflow/core/grappler/utils:graph_view_test PASSED in 3.3s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 6.8s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.6s //tensorflow/core/grappler/utils:scc_test PASSED in 1.5s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.1s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.6s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.3s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.7s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.9s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 2.7s //tensorflow/core/ir:interfaces_test PASSED in 0.2s //tensorflow/core/ir:ops_test PASSED in 0.2s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.2s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.5s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.1s //tensorflow/core/ir:utility_test PASSED in 0.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 2.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 1.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 2.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 2.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 2.1s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 1.2s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 1.6s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.9s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.6s //tensorflow/core/ir/types:dialect_test PASSED in 0.2s //tensorflow/core/kernels:as_string_op_test PASSED in 1.0s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 1.1s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 0.9s //tensorflow/core/kernels:batch_kernels_test PASSED in 43.2s //tensorflow/core/kernels:bias_op_test PASSED in 1.3s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 0.7s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.5s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.9s //tensorflow/core/kernels:concat_op_test PASSED in 0.6s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:control_flow_ops_test PASSED in 7.8s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 7.3s //tensorflow/core/kernels:count_ops_test PASSED in 0.5s //tensorflow/core/kernels:cross_op_test PASSED in 1.4s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:debug_ops_test PASSED in 1.5s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.5s //tensorflow/core/kernels:deep_conv2d_test PASSED in 1.0s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.6s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 20.3s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 1.3s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.3s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.6s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 0.9s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.3s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.5s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.3s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.3s //tensorflow/core/kernels:encode_wav_op_test PASSED in 3.5s //tensorflow/core/kernels:fingerprint_op_test PASSED in 1.0s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 0.9s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.7s //tensorflow/core/kernels:identity_op_test PASSED in 0.6s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 0.9s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.4s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 1.4s //tensorflow/core/kernels:logging_ops_test PASSED in 1.8s //tensorflow/core/kernels:lookup_ops_test PASSED in 1.5s //tensorflow/core/kernels:loss_test PASSED in 0.1s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 4.2s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 0.6s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.7s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.0s //tensorflow/core/kernels:mfcc_test PASSED in 0.7s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.7s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.5s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.4s //tensorflow/core/kernels:ops_util_test PASSED in 0.7s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 1.4s //tensorflow/core/kernels:parse_tensor_test PASSED in 0.9s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.6s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.6s //tensorflow/core/kernels:quantize_op_test PASSED in 0.8s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.5s //tensorflow/core/kernels:quantized_add_op_test PASSED in 11.7s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.9s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 1.1s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 0.5s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.1s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 0.5s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 1.8s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 0.8s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 0.8s //tensorflow/core/kernels:ragged_range_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.4s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 0.5s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 1.4s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 0.8s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.2s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:range_sampler_test PASSED in 0.7s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.7s //tensorflow/core/kernels:requantization_range_op_test PASSED in 0.5s //tensorflow/core/kernels:requantize_op_test PASSED in 0.6s //tensorflow/core/kernels:resource_ops_test PASSED in 0.5s //tensorflow/core/kernels:restore_op_test PASSED in 1.2s //tensorflow/core/kernels:restore_v2_op_test PASSED in 0.5s //tensorflow/core/kernels:reverse_op_test PASSED in 1.9s //tensorflow/core/kernels:roll_op_test PASSED in 0.6s //tensorflow/core/kernels:save_op_test PASSED in 0.8s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.9s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.8s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:scatter_op_test PASSED in 1.4s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 8.1s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.3s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 0.7s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.8s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.6s //tensorflow/core/kernels:shape_ops_test PASSED in 0.5s //tensorflow/core/kernels:slice_op_test PASSED in 0.5s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.4s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.8s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 0.8s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.7s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 1.8s //tensorflow/core/kernels:spectrogram_test PASSED in 0.1s //tensorflow/core/kernels:split_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.6s //tensorflow/core/kernels:string_format_op_test PASSED in 0.8s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 0.8s //tensorflow/core/kernels:string_split_op_test PASSED in 0.6s //tensorflow/core/kernels:substr_op_test PASSED in 0.5s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.8s //tensorflow/core/kernels:summary_image_op_test PASSED in 1.7s //tensorflow/core/kernels:summary_op_test PASSED in 1.4s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.5s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.2s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.1s //tensorflow/core/kernels:tensor_map_test PASSED in 0.2s //tensorflow/core/kernels:training_ops_test PASSED in 0.9s //tensorflow/core/kernels:transpose_util_test PASSED in 0.5s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 1.6s //tensorflow/core/kernels:unique_op_test PASSED in 0.5s //tensorflow/core/kernels:variable_ops_test PASSED in 1.6s //tensorflow/core/kernels:while_op_test PASSED in 0.7s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.6s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.2s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 1.7s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 1.6s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 2.7s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 1.0s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 3.2s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 15.2s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 2.9s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data:get_options_op_test PASSED in 0.9s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 1.4s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 2.7s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 2.1s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 2.3s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 3.0s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 1.8s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.9s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 3.3s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 2.7s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 2.1s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 3.8s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 15.0s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 2.3s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.9s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 1.0s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 15.1s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 1.6s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 2.4s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.6s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 0.7s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 2.0s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 6.1s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 2.3s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 4.9s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 2.9s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.1s //tensorflow/core/kernels/sparse:kernels_test PASSED in 1.3s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.2s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.4s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.4s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 2.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 0.9s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 0.6s //tensorflow/core/lib/db:sqlite_test PASSED in 0.1s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 1.0s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.9s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 0.7s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.8s //tensorflow/core/ops:ops_math_grad_test PASSED in 3.6s //tensorflow/core/ops:ops_tests PASSED in 1.0s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.6s //tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.2s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.3s //tensorflow/core/platform:env_test PASSED in 2.7s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 0.4s //tensorflow/core/platform:platform_strings_test PASSED in 0.4s //tensorflow/core/platform:ram_file_system_test PASSED in 41.2s //tensorflow/core/platform:resource_loader_test PASSED in 0.1s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.1s //tensorflow/core/platform:vmodule_test PASSED in 0.2s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.1s //tensorflow/core/profiler/convert:dcn_analysis_test PASSED in 0.1s //tensorflow/core/profiler/convert:dcn_utils_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.6s //tensorflow/core/profiler/convert:repository_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_dcn_collective_stats_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.8s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.6s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.2s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.2s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 0.8s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 1.1s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 0.5s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 0.9s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 0.6s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.1s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.1s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.1s //tensorflow/core/runtime_fallback/util:type_util_test PASSED in 0.1s //tensorflow/core/summary:schema_test PASSED in 0.2s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.2s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.1s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 6.2s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 9.3s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 7.2s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.5s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.6s //tensorflow/core/tfrt/graph_executor:config_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/attribute:attribute_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/bytecode:bytecode_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:executable_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:function_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:kernel_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:span_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/interpreter:context_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:future_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:interpreter_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:register_span_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:value_test PASSED in 0.1s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_concurrent_work_queue_test PASSED in 0.4s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_test PASSED in 0.6s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_util_test PASSED in 0.1s //tensorflow/core/tfrt/runtime:channel_test PASSED in 1.1s //tensorflow/core/tfrt/runtime:tf_threadpool_concurrent_work_queue_test PASSED in 0.3s //tensorflow/core/tfrt/runtime:work_queue_interface_test PASSED in 0.1s //tensorflow/core/tfrt/utils:graph_partition_test PASSED in 2.4s //tensorflow/core/transforms:eval_utils_test PASSED in 1.6s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.7s //tensorflow/core/util:bcast_test PASSED in 1.7s //tensorflow/core/util:command_line_flags_test PASSED in 0.7s //tensorflow/core/util:debug_data_dumper_test PASSED in 0.7s //tensorflow/core/util:debug_events_writer_test PASSED in 0.3s //tensorflow/core/util:dump_graph_test PASSED in 0.7s //tensorflow/core/util:equal_graph_def_test PASSED in 1.2s //tensorflow/core/util:events_writer_test PASSED in 3.2s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.9s //tensorflow/core/util:example_proto_helper_test PASSED in 0.8s //tensorflow/core/util:exec_on_stall_test PASSED in 2.1s //tensorflow/core/util:fake_clock_env_test PASSED in 2.0s //tensorflow/core/util:incremental_barrier_test PASSED in 1.3s //tensorflow/core/util:matmul_bcast_test PASSED in 1.0s //tensorflow/core/util:memmapped_file_system_test PASSED in 0.8s //tensorflow/core/util:mkl_heuristics_test PASSED in 0.9s //tensorflow/core/util:overflow_test PASSED in 0.1s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 1.9s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.6s //tensorflow/core/util:reffed_status_callback_test PASSED in 0.9s //tensorflow/core/util:reporter_test PASSED in 0.8s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.9s //tensorflow/core/util:semver_test PASSED in 0.8s //tensorflow/core/util:stat_summarizer_test PASSED in 1.1s //tensorflow/core/util:strided_slice_op_test PASSED in 0.9s //tensorflow/core/util:tensor_format_test PASSED in 0.8s //tensorflow/core/util:tensor_slice_reader_test PASSED in 0.9s //tensorflow/core/util:tensor_slice_set_test PASSED in 0.9s //tensorflow/core/util:tensor_slice_util_test PASSED in 0.8s //tensorflow/core/util:tensor_slice_writer_test PASSED in 2.1s //tensorflow/core/util:work_sharder_test PASSED in 1.5s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.1s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.9s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.9s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.3s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.2s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 32.9s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.1s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:decompose_controlflow.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:dtensor_collective_type_lowering.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 15.0s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 1.7s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 2.0s //tensorflow/dtensor/mlir/tests:spmd_fft.mlir.test PASSED in 2.0s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 16.0s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 1.4s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 1.8s //tensorflow/dtensor/python/tests:array_ops_test_cpu PASSED in 27.7s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 23.1s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 18.6s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 10.6s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 48.7s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 15.4s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 19.0s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 16.8s //tensorflow/dtensor/python/tests:variable_test_cpu PASSED in 19.9s //tensorflow/dtensor/tests:dtensor_operation_test PASSED in 29.7s //tensorflow/dtensor/tests:executable_manager_test PASSED in 37.6s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.1s //tensorflow/dtensor/tests:slice_util_test PASSED in 0.1s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 8.5s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.2s //tensorflow/examples/adding_an_op:fact_test PASSED in 22.0s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 21.9s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 25.1s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 35.7s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 24.9s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 26.3s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 23.7s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 27.2s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 24.6s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 24.4s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 1.8s //tensorflow/examples/speech_commands:models_test PASSED in 32.4s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.3s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.7s //tensorflow/js:ts_op_gen_test PASSED in 0.1s //tensorflow/python/autograph/converters:asserts_test PASSED in 12.4s //tensorflow/python/autograph/converters:break_statements_test PASSED in 12.6s //tensorflow/python/autograph/converters:call_trees_test PASSED in 12.5s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 13.2s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 10.9s //tensorflow/python/autograph/converters:control_flow_test PASSED in 18.6s //tensorflow/python/autograph/converters:directives_test PASSED in 9.8s //tensorflow/python/autograph/converters:functions_test PASSED in 10.1s //tensorflow/python/autograph/converters:lists_test PASSED in 11.7s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 10.5s //tensorflow/python/autograph/converters:return_statements_test PASSED in 51.0s //tensorflow/python/autograph/converters:slices_test PASSED in 9.6s //tensorflow/python/autograph/converters:variables_test PASSED in 12.6s //tensorflow/python/autograph/core:converter_test PASSED in 9.5s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 25.2s //tensorflow/python/autograph/impl:api_test PASSED in 29.6s //tensorflow/python/autograph/impl:conversion_test PASSED in 10.6s //tensorflow/python/autograph/lang:special_functions_test PASSED in 26.4s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 10.6s //tensorflow/python/autograph/operators:control_flow_test PASSED in 20.7s //tensorflow/python/autograph/operators:data_structures_test PASSED in 10.9s //tensorflow/python/autograph/operators:exceptions_test PASSED in 9.4s //tensorflow/python/autograph/operators:logical_test PASSED in 10.9s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 29.1s //tensorflow/python/autograph/operators:slices_test PASSED in 14.8s //tensorflow/python/autograph/operators:variables_test PASSED in 10.4s //tensorflow/python/autograph/pyct:anno_test PASSED in 26.9s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 10.2s //tensorflow/python/autograph/pyct:cache_test PASSED in 11.3s //tensorflow/python/autograph/pyct:cfg_test PASSED in 12.5s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 11.3s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 10.6s //tensorflow/python/autograph/pyct:loader_test PASSED in 9.2s //tensorflow/python/autograph/pyct:naming_test PASSED in 10.9s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 13.5s //tensorflow/python/autograph/pyct:parser_test PASSED in 13.2s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 9.4s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 9.0s //tensorflow/python/autograph/pyct:templates_test PASSED in 10.8s //tensorflow/python/autograph/pyct:transformer_test PASSED in 10.0s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 37.0s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 11.8s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 13.0s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 13.6s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 10.5s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 9.2s //tensorflow/python/autograph/tests:assertion_test PASSED in 23.4s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 24.0s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 72.5s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 25.7s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 22.9s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 39.2s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 68.0s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 21.8s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 23.0s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 31.0s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 32.8s //tensorflow/python/autograph/tests:datasets_test PASSED in 44.3s //tensorflow/python/autograph/tests:early_return_test PASSED in 27.7s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 44.1s //tensorflow/python/autograph/tests:generator_test PASSED in 24.4s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 25.0s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 111.8s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 40.5s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 49.7s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 27.4s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 39.6s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 46.6s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 67.1s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 89.9s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 24.5s //tensorflow/python/autograph/utils:context_managers_test PASSED in 9.8s //tensorflow/python/autograph/utils:misc_test PASSED in 10.4s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 10.6s //tensorflow/python/autograph/utils:tensors_test PASSED in 9.0s //tensorflow/python/checkpoint:benchmarks_test PASSED in 12.2s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 19.6s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 18.3s //tensorflow/python/checkpoint:checkpoint_test PASSED in 27.3s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 19.7s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 17.0s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 12.5s //tensorflow/python/checkpoint:restore_test PASSED in 23.2s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 9.7s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 11.5s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 21.9s //tensorflow/python/checkpoint:trackable_view_test PASSED in 13.7s //tensorflow/python/client:device_lib_test_cpu PASSED in 11.7s //tensorflow/python/client:events_writer_test PASSED in 10.6s //tensorflow/python/client:session_benchmark_cpu PASSED in 15.9s //tensorflow/python/client:session_list_devices_test PASSED in 24.4s //tensorflow/python/client:session_partial_run_test PASSED in 29.6s //tensorflow/python/client:timeline_test_cpu PASSED in 10.8s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 14.0s //tensorflow/python/compat:compat_test PASSED in 9.3s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 14.0s //tensorflow/python/compiler/mlir:mlir_test PASSED in 12.0s //tensorflow/python/compiler/tensorrt:trt_convert_test_cpu PASSED in 14.1s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 12.7s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 11.9s //tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 15.3s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 17.6s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 25.0s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 24.7s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 14.3s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 12.6s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 25.6s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 15.7s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 11.8s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 11.2s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 21.6s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 13.4s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 28.1s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 13.3s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 31.5s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 14.7s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 26.4s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 12.8s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 16.1s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 12.4s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 27.1s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 15.0s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 30.9s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 31.4s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 14.6s //tensorflow/python/data/benchmarks:batch_benchmark PASSED in 15.7s //tensorflow/python/data/benchmarks:filter_benchmark PASSED in 13.0s //tensorflow/python/data/benchmarks:from_tensor_slices_benchmark PASSED in 14.3s //tensorflow/python/data/benchmarks:interleave_benchmark PASSED in 17.6s //tensorflow/python/data/benchmarks:list_files_benchmark PASSED in 11.7s //tensorflow/python/data/benchmarks:map_benchmark PASSED in 13.7s //tensorflow/python/data/benchmarks:meta_benchmark PASSED in 10.2s //tensorflow/python/data/benchmarks:prefetch_benchmark PASSED in 13.9s //tensorflow/python/data/benchmarks:range_benchmark PASSED in 15.2s //tensorflow/python/data/experimental/benchmarks:autotune_benchmark PASSED in 10.3s //tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark PASSED in 13.0s //tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 13.2s //tensorflow/python/data/experimental/benchmarks:map_defun_benchmark PASSED in 14.2s //tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 15.6s //tensorflow/python/data/experimental/benchmarks:optimize_benchmark PASSED in 27.1s //tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 11.2s //tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 12.4s //tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 15.9s //tensorflow/python/data/experimental/benchmarks:unbatch_benchmark PASSED in 11.3s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 40.0s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 11.3s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 13.0s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 24.3s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 25.8s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 17.6s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 20.5s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 32.5s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 57.6s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 11.3s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 28.5s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 10.4s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 94.7s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 11.4s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 44.7s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 12.6s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 14.6s //tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 12.4s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 15.0s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 30.0s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 24.6s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 10.9s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 12.5s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 10.9s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 12.7s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 35.8s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 75.9s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 12.0s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 45.4s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 16.8s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 33.6s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 22.0s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 14.4s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 22.9s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 16.0s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 12.0s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 11.4s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 20.0s //tensorflow/python/data/kernel_tests:cache_test PASSED in 47.1s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 17.3s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 20.0s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 32.0s //tensorflow/python/data/kernel_tests:counter_test PASSED in 51.6s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 10.9s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 28.6s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 27.5s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 10.0s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 27.4s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 42.9s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 19.0s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 14.9s //tensorflow/python/data/kernel_tests:io_test PASSED in 44.0s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 44.4s //tensorflow/python/data/kernel_tests:len_test PASSED in 10.7s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 14.6s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 12.6s //tensorflow/python/data/kernel_tests:options_test PASSED in 16.6s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 20.2s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 36.9s //tensorflow/python/data/kernel_tests:random_test PASSED in 25.6s //tensorflow/python/data/kernel_tests:range_test PASSED in 51.9s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 15.7s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 29.0s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 37.5s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 25.6s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 61.4s //tensorflow/python/data/util:convert_test PASSED in 13.1s //tensorflow/python/data/util:nest_test PASSED in 10.5s //tensorflow/python/data/util:options_test PASSED in 11.3s //tensorflow/python/data/util:random_seed_test PASSED in 10.7s //tensorflow/python/data/util:sparse_test PASSED in 11.8s //tensorflow/python/data/util:structure_test PASSED in 14.1s //tensorflow/python/data/util:traverse_test PASSED in 11.3s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 12.2s //tensorflow/python/debug/cli:cli_config_test PASSED in 9.8s //tensorflow/python/debug/cli:cli_shared_test PASSED in 10.5s //tensorflow/python/debug/cli:command_parser_test PASSED in 9.7s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 9.4s //tensorflow/python/debug/cli:evaluator_test PASSED in 11.0s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 12.9s //tensorflow/python/debug/cli:readline_ui_test PASSED in 10.4s //tensorflow/python/debug/cli:tensor_format_test PASSED in 10.8s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 14.9s //tensorflow/python/debug/lib:common_test PASSED in 12.7s //tensorflow/python/debug/lib:debug_data_test PASSED in 10.3s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 14.6s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 11.2s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 11.1s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 13.2s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 13.2s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 14.1s //tensorflow/python/debug/lib:debug_utils_test PASSED in 22.4s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 23.4s //tensorflow/python/debug/lib:profiling_test PASSED in 8.6s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 43.1s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 15.4s //tensorflow/python/debug/lib:source_utils_test PASSED in 15.2s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 13.6s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 24.2s //tensorflow/python/debug/wrappers:framework_test PASSED in 21.6s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 14.3s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 14.2s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 16.0s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 13.3s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 25.0s //tensorflow/python/distribute:collective_util_test PASSED in 12.3s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 38.4s //tensorflow/python/distribute:combinations_test_cpu PASSED in 23.5s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 14.8s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 18.6s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 19.2s //tensorflow/python/distribute:device_util_test_cpu PASSED in 16.9s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 21.5s //tensorflow/python/distribute:distribute_lib_test PASSED in 16.6s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 23.6s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 16.7s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 64.6s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 43.6s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 28.8s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 16.8s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 11.4s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 25.9s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 19.9s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 12.1s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 41.8s //tensorflow/python/distribute:multi_worker_util_test PASSED in 8.6s //tensorflow/python/distribute:numpy_dataset_test PASSED in 10.6s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 26.3s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 10.9s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 47.8s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 39.1s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 27.7s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 35.4s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 30.5s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 12.4s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 11.3s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 15.1s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 13.1s //tensorflow/python/distribute:sharded_variable_test PASSED in 26.8s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 11.6s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 58.5s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 14.3s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 22.4s //tensorflow/python/distribute:test_util_test_cpu PASSED in 24.3s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 16.2s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 20.9s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 16.4s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 14.3s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 15.1s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 9.0s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 11.8s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 11.4s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 11.9s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 10.1s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 10.9s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 13.1s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 64.5s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 15.4s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 31.9s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 25.5s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 75.8s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 15.1s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 71.1s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 79.5s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 16.1s //tensorflow/python/eager:backprop_test_cpu PASSED in 152.3s //tensorflow/python/eager:benchmarks_test_cpu PASSED in 10.5s //tensorflow/python/eager:cancellation_test_cpu PASSED in 9.9s //tensorflow/python/eager:context_test_cpu PASSED in 14.9s //tensorflow/python/eager:core_test_cpu PASSED in 14.4s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 51.9s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 11.4s //tensorflow/python/eager:lift_to_graph_test PASSED in 47.8s //tensorflow/python/eager:monitoring_test_cpu PASSED in 12.7s //tensorflow/python/eager:ops_test_cpu PASSED in 15.0s //tensorflow/python/eager:profiler_client_test PASSED in 11.2s //tensorflow/python/eager:profiler_test_cpu PASSED in 9.9s //tensorflow/python/eager:pywrap_tfe_test PASSED in 30.5s //tensorflow/python/eager:record_test PASSED in 14.6s //tensorflow/python/eager:remote_benchmarks_test_cpu PASSED in 13.0s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 11.2s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 9.8s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 209.3s //tensorflow/python/eager:tensor_test_cpu PASSED in 20.3s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 14.6s //tensorflow/python/eager:wrap_function_test PASSED in 15.9s //tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu PASSED in 45.5s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 13.5s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 13.4s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 16.1s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 13.8s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 14.9s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 18.7s //tensorflow/python/eager/polymorphic_function:concrete_function_test_cpu PASSED in 14.6s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 9.4s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 13.8s //tensorflow/python/eager/polymorphic_function:tracing_compilation_test PASSED in 27.8s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 15.7s //tensorflow/python/feature_column:serialization_test PASSED in 16.4s //tensorflow/python/framework:auto_control_deps_test PASSED in 55.0s //tensorflow/python/framework:c_api_util_test PASSED in 56.5s //tensorflow/python/framework:common_shapes_test PASSED in 21.4s //tensorflow/python/framework:composite_tensor_test PASSED in 13.7s //tensorflow/python/framework:config_test_2gpu PASSED in 21.1s //tensorflow/python/framework:config_test_cpu PASSED in 23.9s //tensorflow/python/framework:constant_op_test PASSED in 11.8s //tensorflow/python/framework:device_spec_test PASSED in 12.4s //tensorflow/python/framework:device_test PASSED in 12.3s //tensorflow/python/framework:dtypes_test PASSED in 34.4s //tensorflow/python/framework:error_interpolation_test PASSED in 10.7s //tensorflow/python/framework:errors_test PASSED in 11.7s //tensorflow/python/framework:extension_type_field_test PASSED in 15.8s //tensorflow/python/framework:extension_type_test PASSED in 23.9s //tensorflow/python/framework:file_system_test PASSED in 10.1s //tensorflow/python/framework:flexible_dtypes_test PASSED in 104.9s //tensorflow/python/framework:function_def_to_graph_test PASSED in 21.6s //tensorflow/python/framework:graph_building_benchmark_cpu PASSED in 11.8s //tensorflow/python/framework:graph_util_test PASSED in 13.5s //tensorflow/python/framework:immutable_dict_test PASSED in 10.2s //tensorflow/python/framework:importer_test PASSED in 16.4s //tensorflow/python/framework:indexed_slices_test PASSED in 10.1s //tensorflow/python/framework:kernels_test PASSED in 13.9s //tensorflow/python/framework:meta_graph_test PASSED in 13.0s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 13.0s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.2s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 3.6s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 31.2s //tensorflow/python/framework:op_def_library_test PASSED in 14.4s //tensorflow/python/framework:op_def_util_test PASSED in 10.9s //tensorflow/python/framework:ops_enable_eager_test PASSED in 3.9s //tensorflow/python/framework:ops_test PASSED in 26.6s //tensorflow/python/framework:proto_test PASSED in 20.7s //tensorflow/python/framework:py_context_manager_test PASSED in 31.8s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 12.4s //tensorflow/python/framework:python_api_info_test PASSED in 15.6s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 13.0s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 4.9s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.1s //tensorflow/python/framework:python_op_gen_test PASSED in 0.4s //tensorflow/python/framework:python_tensor_converter_test PASSED in 13.1s //tensorflow/python/framework:random_seed_test PASSED in 9.5s //tensorflow/python/framework:registry_test PASSED in 10.4s //tensorflow/python/framework:smart_cond_test PASSED in 13.5s //tensorflow/python/framework:sparse_tensor_test PASSED in 15.5s //tensorflow/python/framework:subscribe_test PASSED in 15.7s //tensorflow/python/framework:tensor_shape_test PASSED in 12.4s //tensorflow/python/framework:tensor_test PASSED in 10.5s //tensorflow/python/framework:tensor_util_test PASSED in 15.8s //tensorflow/python/framework:test_combinations_test PASSED in 9.8s //tensorflow/python/framework:test_util_test_cpu PASSED in 26.5s //tensorflow/python/framework:tf2_test PASSED in 48.4s //tensorflow/python/framework:traceable_stack_test PASSED in 16.3s //tensorflow/python/framework:type_spec_test PASSED in 13.0s //tensorflow/python/framework:versions_test PASSED in 10.0s //tensorflow/python/framework:weak_tensor_test PASSED in 14.5s //tensorflow/python/framework/experimental:graph_building_test_cpu PASSED in 49.3s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 16.7s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 45.7s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 22.5s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 21.9s //tensorflow/python/grappler:cost_analyzer_test PASSED in 11.3s //tensorflow/python/grappler:datasets_test PASSED in 28.9s //tensorflow/python/grappler:item_test PASSED in 13.9s //tensorflow/python/grappler:memory_optimizer_test PASSED in 23.7s //tensorflow/python/grappler:model_analyzer_test PASSED in 14.3s //tensorflow/python/grappler:remapper_test_cpu PASSED in 30.2s //tensorflow/python/grappler:tf_optimizer_test PASSED in 27.8s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 17.1s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 25.8s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 48.4s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 14.5s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 22.1s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 15.7s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 23.5s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests:template_test PASSED in 23.2s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 23.3s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 14.5s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 10.9s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 14.2s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 20.9s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 37.8s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 15.7s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 28.6s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 28.6s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 13.7s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 16.9s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 14.2s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 11.9s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 10.5s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 15.0s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 14.7s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 18.2s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 16.2s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 19.5s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 22.1s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 14.6s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 21.2s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 15.6s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 28.9s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 67.3s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 12.9s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 23.9s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 22.4s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 70.4s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 113.7s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 10.7s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 9.6s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 11.9s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 16.9s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 16.6s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 16.8s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 20.2s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 23.0s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 11.5s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 33.3s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 25.0s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 12.6s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 11.7s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 20.9s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 18.4s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 17.8s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 14.4s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 52.7s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 13.9s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 37.2s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 29.4s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 29.6s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 28.3s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 16.1s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 13.0s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 13.0s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 10.1s //tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 14.5s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 17.1s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 13.1s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 10.7s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 12.3s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 15.7s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 32.9s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 17.2s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 35.6s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 12.9s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 23.0s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 14.7s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 14.0s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 14.6s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 13.6s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 57.5s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 31.5s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 19.7s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 14.6s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 17.3s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 14.4s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 11.7s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 15.3s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 19.2s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu PASSED in 49.4s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 31.1s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 159.6s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 24.1s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 11.0s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 97.9s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 11.0s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 39.3s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 15.3s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 15.1s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 34.4s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 22.8s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 22.2s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 134.6s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 13.5s //tensorflow/python/kernel_tests/proto:decode_proto_op_test PASSED in 23.7s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 10.6s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 12.8s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 10.8s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 18.2s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 17.4s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 15.6s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 21.3s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 9.9s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 20.7s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 146.7s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 16.8s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 13.3s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 28.4s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 24.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 17.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 15.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 16.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 42.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 19.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 11.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 14.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 16.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 12.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 22.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 57.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 24.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 11.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 14.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 84.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 11.8s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 14.8s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 15.7s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 22.3s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 11.8s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 16.2s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 33.1s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 10.2s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 10.1s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 11.6s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 17.7s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 10.2s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 11.2s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 22.9s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 12.3s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 31.7s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 14.6s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 13.5s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 11.5s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 32.0s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 11.7s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 20.6s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 13.0s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 14.5s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 18.6s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 89.7s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 45.7s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 15.9s //tensorflow/python/lib/io:file_io_test PASSED in 12.5s //tensorflow/python/lib/io:tf_record_test PASSED in 16.9s //tensorflow/python/module:module_test PASSED in 15.7s //tensorflow/python/ops:array_grad_test_cpu PASSED in 12.7s //tensorflow/python/ops:array_ops_shape_test PASSED in 11.2s //tensorflow/python/ops:array_ops_test PASSED in 12.9s //tensorflow/python/ops:autograph_ops_test PASSED in 9.9s //tensorflow/python/ops:batch_norm_benchmark_cpu PASSED in 15.1s //tensorflow/python/ops:bincount_ops_test_cpu PASSED in 15.6s //tensorflow/python/ops:bitwise_ops_test_cpu PASSED in 11.0s //tensorflow/python/ops:clip_ops_test PASSED in 11.6s //tensorflow/python/ops:clustering_ops_test PASSED in 26.7s //tensorflow/python/ops:collective_ops_benchmark_cpu PASSED in 10.4s //tensorflow/python/ops:collective_ops_gpu_test_cpu PASSED in 11.2s //tensorflow/python/ops:collective_ops_test PASSED in 24.1s //tensorflow/python/ops:collective_ops_xla_test PASSED in 11.7s //tensorflow/python/ops:compiled_collective_ops_gpu_test_2gpu PASSED in 11.1s //tensorflow/python/ops:compiled_collective_ops_gpu_test_cpu PASSED in 11.5s //tensorflow/python/ops:concat_benchmark_cpu PASSED in 14.2s //tensorflow/python/ops:control_flow_ops_benchmark_cpu PASSED in 13.0s //tensorflow/python/ops:control_flow_v2_enable_test PASSED in 11.2s //tensorflow/python/ops:control_flow_v2_toggles_test PASSED in 15.1s //tensorflow/python/ops:dequantize_op_test PASSED in 13.0s //tensorflow/python/ops:embedding_ops_test_cpu PASSED in 14.5s //tensorflow/python/ops:factory_ops_test_cpu PASSED in 10.5s //tensorflow/python/ops:functional_ops_test PASSED in 13.9s //tensorflow/python/ops:gradient_checker_v2_test_cpu PASSED in 44.1s //tensorflow/python/ops:gradients_test_cpu PASSED in 20.8s //tensorflow/python/ops:init_ops_test_cpu PASSED in 16.2s //tensorflow/python/ops:init_ops_v2_test_cpu PASSED in 17.4s //tensorflow/python/ops:lookup_ops_async_checkpoint_test PASSED in 12.8s //tensorflow/python/ops:math_grad_test_cpu PASSED in 58.2s //tensorflow/python/ops:math_ops_linspace_test_cpu PASSED in 29.8s //tensorflow/python/ops:math_ops_test_cpu PASSED in 29.3s //tensorflow/python/ops:matmul_benchmark_cpu PASSED in 11.4s //tensorflow/python/ops:nn_grad_test_cpu PASSED in 18.7s //tensorflow/python/ops:nn_loss_scaling_utilities_test PASSED in 13.3s //tensorflow/python/ops:nn_test_cpu PASSED in 87.6s //tensorflow/python/ops:nn_xent_test_cpu PASSED in 15.1s //tensorflow/python/ops:op_selector_test PASSED in 8.9s //tensorflow/python/ops:quantized_conv_ops_test PASSED in 11.5s //tensorflow/python/ops:quantized_ops_test PASSED in 13.4s //tensorflow/python/ops:raw_ops_test_cpu PASSED in 9.9s //tensorflow/python/ops:rnn_grad_test_cpu PASSED in 10.0s //tensorflow/python/ops:script_ops_test PASSED in 16.6s //tensorflow/python/ops:sort_ops_test PASSED in 11.6s //tensorflow/python/ops:sparse_bincount_ops_test_cpu PASSED in 16.9s //tensorflow/python/ops:sparse_ops_test PASSED in 18.8s //tensorflow/python/ops:split_benchmark_cpu PASSED in 9.6s //tensorflow/python/ops:tensor_array_ops_test PASSED in 9.5s //tensorflow/python/ops:transpose_benchmark_cpu PASSED in 13.5s //tensorflow/python/ops:variable_spec_test PASSED in 13.4s //tensorflow/python/ops:weak_tensor_array_ops_test PASSED in 10.3s //tensorflow/python/ops:weak_tensor_constant_op_test PASSED in 30.8s //tensorflow/python/ops:weak_tensor_image_ops_test PASSED in 11.3s //tensorflow/python/ops:weak_tensor_math_ops_test PASSED in 31.0s //tensorflow/python/ops:weak_tensor_nn_test_cpu PASSED in 18.2s //tensorflow/python/ops:weak_tensor_np_array_ops_test PASSED in 42.3s //tensorflow/python/ops:weak_tensor_np_math_ops_test PASSED in 12.0s //tensorflow/python/ops:weak_tensor_ops_test PASSED in 109.3s //tensorflow/python/ops/losses:util_test PASSED in 20.5s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 15.3s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 106.6s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 16.8s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 10.8s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 78.3s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 17.2s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 34.1s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 68.4s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 11.4s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 25.7s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 22.5s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 57.5s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 14.8s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 64.9s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 12.5s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 51.3s //tensorflow/python/ops/ragged:ragged_bincount_ops_test_cpu PASSED in 10.7s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 12.4s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 23.7s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 41.8s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 33.8s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 10.5s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 29.6s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 141.3s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 21.5s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 18.2s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 12.8s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 37.0s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 13.5s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 13.0s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 26.3s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 18.2s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 18.8s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 18.1s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 18.8s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 40.1s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 42.3s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 14.0s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 30.5s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 13.1s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 30.0s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 12.7s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 11.7s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 76.4s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 37.5s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 23.0s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 12.3s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 12.4s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 13.4s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 17.4s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 13.1s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 52.5s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 19.6s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 15.3s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 14.4s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 65.0s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 74.5s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 9.8s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 63.9s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 23.2s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 39.7s //tensorflow/python/ops/ragged:row_partition_test PASSED in 28.6s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 8.9s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 13.7s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 61.2s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 53.7s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 14.8s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 50.1s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 20.5s //tensorflow/python/platform:benchmark_test PASSED in 10.5s //tensorflow/python/platform:build_info_test PASSED in 9.5s //tensorflow/python/platform:resource_loader_test PASSED in 3.2s //tensorflow/python/profiler:pprof_profiler_test PASSED in 10.6s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 26.6s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 10.6s //tensorflow/python/profiler:profiler_test_cpu PASSED in 21.9s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 10.7s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 10.4s //tensorflow/python/profiler:tfprof_logger_test PASSED in 13.5s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 12.4s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 26.3s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 23.7s //tensorflow/python/saved_model:fingerprinting_test PASSED in 12.0s //tensorflow/python/saved_model:keras_injection_test PASSED in 23.1s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 23.9s //tensorflow/python/saved_model:loader_test PASSED in 19.3s //tensorflow/python/saved_model:method_name_updater_test PASSED in 10.8s //tensorflow/python/saved_model:metrics_test PASSED in 16.4s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 16.9s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 10.1s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 10.3s //tensorflow/python/saved_model:revived_types_test PASSED in 9.5s //tensorflow/python/saved_model:save_context_test PASSED in 14.6s //tensorflow/python/saved_model:save_test PASSED in 34.9s //tensorflow/python/saved_model:saved_model_test PASSED in 23.0s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 13.2s //tensorflow/python/saved_model:simple_save_test PASSED in 10.7s //tensorflow/python/saved_model:tracing_utils_test PASSED in 13.3s //tensorflow/python/saved_model:utils_test PASSED in 15.1s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 12.9s //tensorflow/python/saved_model/model_utils:export_test PASSED in 16.7s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 9.5s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 20.3s //tensorflow/python/saved_model/registration:registration_test PASSED in 11.6s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 30.1s //tensorflow/python/saved_model/tests:variable_wrapper_test PASSED in 11.3s //tensorflow/python/summary:plugin_asset_test PASSED in 9.8s //tensorflow/python/summary:summary_iterator_test PASSED in 13.4s //tensorflow/python/summary:summary_test PASSED in 12.1s //tensorflow/python/summary:summary_v2_test PASSED in 16.9s //tensorflow/python/summary/writer:writer_test PASSED in 20.0s //tensorflow/python/tools:aot_compiled_test PASSED in 22.6s //tensorflow/python/tools:freeze_graph_test PASSED in 10.3s //tensorflow/python/tools:optimize_for_inference_test PASSED in 10.3s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 25.6s //tensorflow/python/tools:saved_model_cli_test PASSED in 34.4s //tensorflow/python/tools:saved_model_utils_test PASSED in 14.2s //tensorflow/python/tools:strip_unused_test PASSED in 13.8s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 13.8s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 22.4s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 13.7s //tensorflow/python/tools/api/generator2/extractor:parser_test PASSED in 10.2s //tensorflow/python/tools/api/generator2/generator:generator_test PASSED in 1.0s //tensorflow/python/tools/api/generator2/shared:exported_api_test PASSED in 11.9s //tensorflow/python/tpu:bfloat16_test PASSED in 15.2s //tensorflow/python/tpu:feature_column_test PASSED in 18.2s //tensorflow/python/tpu:topology_test PASSED in 10.0s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 13.7s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 28.5s //tensorflow/python/tpu:tpu_infeed_test PASSED in 12.7s //tensorflow/python/tpu:tpu_sharding_test PASSED in 35.4s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 16.8s //tensorflow/python/tpu/client:client_py_test PASSED in 10.2s //tensorflow/python/trackable:autotrackable_test PASSED in 10.7s //tensorflow/python/trackable:base_delegate_test PASSED in 12.3s //tensorflow/python/trackable:base_test PASSED in 25.2s //tensorflow/python/trackable:data_structures_test PASSED in 16.0s //tensorflow/python/trackable:python_state_test PASSED in 14.8s //tensorflow/python/trackable:resource_test PASSED in 10.6s //tensorflow/python/trackable:trackable_utils_test PASSED in 10.4s //tensorflow/python/training:adadelta_test_cpu PASSED in 20.3s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 13.7s //tensorflow/python/training:adagrad_test_cpu PASSED in 17.3s //tensorflow/python/training:adam_test_cpu PASSED in 18.6s //tensorflow/python/training:basic_loops_test_cpu PASSED in 52.0s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 25.8s //tensorflow/python/training:checkpoint_ops_test PASSED in 10.1s //tensorflow/python/training:coordinator_test_cpu PASSED in 18.0s //tensorflow/python/training:device_setter_test_cpu PASSED in 11.5s //tensorflow/python/training:ftrl_test_cpu PASSED in 16.2s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 13.6s //tensorflow/python/training:input_test PASSED in 27.6s //tensorflow/python/training:momentum_test_cpu PASSED in 13.5s //tensorflow/python/training:monitored_session_test PASSED in 32.4s //tensorflow/python/training:moving_averages_test_cpu PASSED in 20.7s //tensorflow/python/training:optimizer_test_cpu PASSED in 14.3s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 12.6s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 13.7s //tensorflow/python/training:quantize_training_test_cpu PASSED in 10.5s //tensorflow/python/training:queue_runner_test_cpu PASSED in 10.5s //tensorflow/python/training:rmsprop_test_cpu PASSED in 30.1s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 17.3s //tensorflow/python/training:saver_test_2gpu PASSED in 43.7s //tensorflow/python/training:saver_test_cpu PASSED in 45.1s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 10.1s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 12.5s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 14.1s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 18.1s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 15.3s //tensorflow/python/training:server_lib_test PASSED in 19.7s //tensorflow/python/training:session_manager_test_cpu PASSED in 78.9s //tensorflow/python/training:slot_creator_test_cpu PASSED in 17.8s //tensorflow/python/training:supervisor_test PASSED in 19.9s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 50.9s //tensorflow/python/training:training_ops_test_cpu PASSED in 12.2s //tensorflow/python/training:training_util_test PASSED in 11.1s //tensorflow/python/training:warm_starting_util_test PASSED in 30.9s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 26.1s //tensorflow/python/training/experimental:loss_scale_test PASSED in 41.1s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 10.5s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 11.2s //tensorflow/python/util:compat_test PASSED in 10.5s //tensorflow/python/util:decorator_utils_test PASSED in 10.7s //tensorflow/python/util:deprecation_test PASSED in 16.4s //tensorflow/python/util:dispatch_test PASSED in 25.5s //tensorflow/python/util:example_parser_configuration_test PASSED in 11.1s //tensorflow/python/util:fast_module_type_test PASSED in 19.6s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 10.0s //tensorflow/python/util:function_utils_test PASSED in 46.4s //tensorflow/python/util:keyword_args_test PASSED in 11.5s //tensorflow/python/util:lazy_loader_test PASSED in 11.3s //tensorflow/python/util:lock_util_test PASSED in 29.4s //tensorflow/python/util:module_wrapper_test PASSED in 13.8s //tensorflow/python/util:nest_test PASSED in 32.9s //tensorflow/python/util:object_identity_test PASSED in 18.6s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 4.4s //tensorflow/python/util:serialization_test PASSED in 9.9s //tensorflow/python/util:tf_contextlib_test PASSED in 10.2s //tensorflow/python/util:tf_decorator_test PASSED in 9.8s //tensorflow/python/util:tf_export_test PASSED in 14.3s //tensorflow/python/util:tf_inspect_test PASSED in 11.8s //tensorflow/python/util:tf_should_use_test PASSED in 12.0s //tensorflow/python/util:tf_stack_test PASSED in 9.8s //tensorflow/python/util:traceback_utils_test PASSED in 11.4s //tensorflow/python/util:type_annotations_test PASSED in 10.4s //tensorflow/python/util:variable_utils_test PASSED in 17.2s //tensorflow/python/util:vlog_test PASSED in 15.7s //tensorflow/python/util/protobuf:protobuf_compare_test PASSED in 4.6s //tensorflow/tools/api/tests:module_test PASSED in 27.6s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.4s //tensorflow/tools/common:public_api_test PASSED in 2.9s //tensorflow/tools/common:traverse_test PASSED in 3.3s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 10.0s //tensorflow/tools/compatibility:ast_edits_test PASSED in 12.7s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 26.0s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 26.1s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 15.0s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 10.0s //tensorflow/tools/docs:tf_doctest_test PASSED in 1.5s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.5s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 1.5s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 1.9s //tensorflow/tools/graph_transforms:transforms_test PASSED in 4.7s //tensorflow/tools/proto_splitter:merge_test PASSED in 0.2s //tensorflow/tools/proto_splitter:split_graph_def_test PASSED in 15.0s //tensorflow/tools/proto_splitter:split_test PASSED in 10.5s //tensorflow/tools/proto_splitter:util_test PASSED in 13.6s //tensorflow/tools/proto_splitter/cc:composable_splitter_test PASSED in 0.3s //tensorflow/tools/proto_splitter/cc:graph_def_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:saved_model_splitter_test PASSED in 0.8s //tensorflow/tools/proto_splitter/cc:util_test PASSED in 2.2s //tensorflow/tools/proto_splitter/python:saved_model_test PASSED in 10.2s //tensorflow/tools/proto_splitter/python:test_util_test PASSED in 9.7s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.2s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.5s //tensorflow/tsl/c:tsl_status_test PASSED in 0.2s //tensorflow/tsl/concurrency:async_value_ref_test PASSED in 0.2s //tensorflow/tsl/concurrency:async_value_test PASSED in 0.1s //tensorflow/tsl/concurrency:concurrent_vector_test PASSED in 0.1s //tensorflow/tsl/cuda:cudnn_version_test PASSED in 0.1s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 13.4s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.1s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 5.4s //tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 5.5s //tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test PASSED in 0.5s //tensorflow/tsl/distributed_runtime/rpc:grpc_util_test PASSED in 0.2s //tensorflow/tsl/framework:cancellation_test PASSED in 1.1s //tensorflow/tsl/framework:device_id_utils_test PASSED in 4.4s //tensorflow/tsl/framework/convolution:eigen_spatial_convolutions_test PASSED in 0.1s //tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests PASSED in 0.2s //tensorflow/tsl/lib/hash:crc32c_test PASSED in 0.4s //tensorflow/tsl/lib/histogram:histogram_test PASSED in 0.1s //tensorflow/tsl/lib/io:buffered_file_test PASSED in 0.1s //tensorflow/tsl/lib/io:buffered_inputstream_test PASSED in 0.1s //tensorflow/tsl/lib/io:cache_test PASSED in 0.2s //tensorflow/tsl/lib/io:inputbuffer_test PASSED in 1.4s //tensorflow/tsl/lib/io:inputstream_interface_test PASSED in 0.1s //tensorflow/tsl/lib/io:random_inputstream_test PASSED in 0.2s //tensorflow/tsl/lib/io:record_reader_writer_test PASSED in 0.1s //tensorflow/tsl/lib/io:recordio_test PASSED in 0.2s //tensorflow/tsl/lib/io:table_test PASSED in 3.5s //tensorflow/tsl/lib/io:zlib_buffers_test PASSED in 10.6s //tensorflow/tsl/lib/io/snappy:snappy_test PASSED in 0.3s //tensorflow/tsl/lib/math:math_util_test PASSED in 0.1s //tensorflow/tsl/lib/random:distribution_sampler_test PASSED in 0.5s //tensorflow/tsl/lib/random:philox_random_test PASSED in 0.1s //tensorflow/tsl/lib/random:random_distributions_test PASSED in 19.2s //tensorflow/tsl/lib/random:simple_philox_test PASSED in 0.1s //tensorflow/tsl/lib/random:weighted_picker_test PASSED in 12.4s //tensorflow/tsl/platform:criticality_test PASSED in 0.1s //tensorflow/tsl/platform:ctstring_test PASSED in 0.5s //tensorflow/tsl/platform:denormal_test PASSED in 0.1s //tensorflow/tsl/platform:errors_test PASSED in 0.7s //tensorflow/tsl/platform:fingerprint_test PASSED in 0.1s //tensorflow/tsl/platform:hash_test PASSED in 0.1s //tensorflow/tsl/platform:integral_types_test PASSED in 0.1s //tensorflow/tsl/platform:intrusive_ptr_test PASSED in 0.1s //tensorflow/tsl/platform:logging_test PASSED in 23.7s //tensorflow/tsl/platform:mutex_test PASSED in 0.2s //tensorflow/tsl/platform:net_test PASSED in 0.1s //tensorflow/tsl/platform:numbers_test PASSED in 0.1s //tensorflow/tsl/platform:path_test PASSED in 0.1s //tensorflow/tsl/platform:port_test PASSED in 8.3s //tensorflow/tsl/platform:random_test PASSED in 2.7s //tensorflow/tsl/platform:refcount_test PASSED in 2.3s //tensorflow/tsl/platform:retrying_file_system_test PASSED in 0.2s //tensorflow/tsl/platform:retrying_utils_test PASSED in 0.4s //tensorflow/tsl/platform:scanner_test PASSED in 0.1s //tensorflow/tsl/platform:setround_test PASSED in 0.1s //tensorflow/tsl/platform:stacktrace_handler_test PASSED in 1.7s //tensorflow/tsl/platform:stacktrace_test PASSED in 0.1s //tensorflow/tsl/platform:status_matchers_test PASSED in 0.1s //tensorflow/tsl/platform:status_test PASSED in 0.3s //tensorflow/tsl/platform:statusor_test PASSED in 2.6s //tensorflow/tsl/platform:str_util_test PASSED in 0.2s //tensorflow/tsl/platform:strcat_test PASSED in 1.0s //tensorflow/tsl/platform:stringpiece_test PASSED in 0.6s //tensorflow/tsl/platform:stringprintf_test PASSED in 0.1s //tensorflow/tsl/platform:subprocess_test PASSED in 0.6s //tensorflow/tsl/platform:tstring_test PASSED in 0.1s //tensorflow/tsl/platform:unbounded_work_queue_test PASSED in 1.4s //tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:curl_http_request_test PASSED in 8.3s //tensorflow/tsl/platform/cloud:expiring_lru_cache_test PASSED in 0.9s //tensorflow/tsl/platform/cloud:gcs_dns_cache_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:gcs_file_system_test PASSED in 5.1s //tensorflow/tsl/platform/cloud:gcs_throttle_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:google_auth_provider_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:oauth_client_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:ram_file_block_cache_test PASSED in 3.0s //tensorflow/tsl/platform/cloud:time_util_test PASSED in 0.1s //tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test PASSED in 0.4s //tensorflow/tsl/profiler/convert:trace_container_test PASSED in 0.3s //tensorflow/tsl/profiler/convert:trace_events_to_json_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:xla_op_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xplane_to_trace_events_test PASSED in 0.5s //tensorflow/tsl/profiler/lib:profiler_factory_test PASSED in 0.2s //tensorflow/tsl/profiler/lib:profiler_lock_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:scoped_annotation_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:traceme_encode_test PASSED in 0.6s //tensorflow/tsl/profiler/rpc/client:profiler_client_test PASSED in 3.4s //tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.4s //tensorflow/tsl/profiler/utils:buffer_pool_test PASSED in 0.4s //tensorflow/tsl/profiler/utils:group_events_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:parse_annotation_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:preprocess_xplane_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tf_op_utils_test PASSED in 0.3s //tensorflow/tsl/profiler/utils:timespan_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tpu_xplane_utils_test PASSED in 0.3s //tensorflow/tsl/profiler/utils:xplane_builder_test PASSED in 0.4s //tensorflow/tsl/profiler/utils:xplane_utils_test PASSED in 0.6s //tensorflow/tsl/util:device_name_utils_test PASSED in 0.1s //tensorflow/tsl/util:stats_calculator_test PASSED in 0.3s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 11.3s Stats over 2 runs: max = 11.3s, min = 9.7s, avg = 10.5s, dev = 0.8s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 12.4s Stats over 2 runs: max = 12.4s, min = 11.2s, avg = 11.8s, dev = 0.6s //tensorflow/compiler/xla/tests:conditional_test_cpu PASSED in 8.7s Stats over 2 runs: max = 8.7s, min = 8.3s, avg = 8.5s, dev = 0.2s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 28.1s Stats over 2 runs: max = 28.1s, min = 19.2s, avg = 23.7s, dev = 4.5s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 21.6s Stats over 2 runs: max = 21.6s, min = 20.3s, avg = 21.0s, dev = 0.7s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 28.7s Stats over 2 runs: max = 28.7s, min = 26.5s, avg = 27.6s, dev = 1.1s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 70.2s Stats over 2 runs: max = 70.2s, min = 63.2s, avg = 66.7s, dev = 3.5s //tensorflow/python/data/kernel_tests:window_test PASSED in 73.2s Stats over 2 runs: max = 73.2s, min = 52.2s, avg = 62.7s, dev = 10.5s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 14.7s Stats over 2 runs: max = 14.7s, min = 14.3s, avg = 14.5s, dev = 0.2s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 22.8s Stats over 2 runs: max = 22.8s, min = 22.1s, avg = 22.4s, dev = 0.4s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 21.3s Stats over 2 runs: max = 21.3s, min = 19.0s, avg = 20.1s, dev = 1.1s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 34.9s Stats over 2 runs: max = 34.9s, min = 23.6s, avg = 29.2s, dev = 5.7s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 143.4s Stats over 2 runs: max = 143.4s, min = 69.6s, avg = 106.5s, dev = 36.9s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 12.3s Stats over 2 runs: max = 12.3s, min = 11.4s, avg = 11.8s, dev = 0.5s //tensorflow/python/ops:control_flow_ops_test_cpu PASSED in 41.9s Stats over 2 runs: max = 41.9s, min = 33.8s, avg = 37.8s, dev = 4.0s //tensorflow/compiler/xla/pjrt/distributed:client_server_test FLAKY, failed in 1 out of 2 in 19.7s Stats over 2 runs: max = 19.7s, min = 5.2s, avg = 12.5s, dev = 7.3s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/pjrt/distributed/client_server_test/test_attempts/attempt_1.log //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 12.6s Stats over 3 runs: max = 12.6s, min = 12.4s, avg = 12.4s, dev = 0.1s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 14.0s Stats over 3 runs: max = 14.0s, min = 13.5s, avg = 13.7s, dev = 0.2s //tensorflow/compiler/xla/tests:triangular_solve_test_cpu PASSED in 56.1s Stats over 3 runs: max = 56.1s, min = 52.3s, avg = 54.2s, dev = 1.5s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.5s Stats over 3 runs: max = 0.5s, min = 0.2s, avg = 0.3s, dev = 0.1s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 18.8s Stats over 3 runs: max = 18.8s, min = 15.0s, avg = 17.2s, dev = 1.6s //tensorflow/python/data/kernel_tests:unique_test PASSED in 17.3s Stats over 3 runs: max = 17.3s, min = 15.2s, avg = 16.0s, dev = 1.0s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 24.0s Stats over 3 runs: max = 24.0s, min = 16.7s, avg = 20.5s, dev = 3.0s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 50.3s Stats over 3 runs: max = 50.3s, min = 33.7s, avg = 39.9s, dev = 7.4s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 15.3s Stats over 3 runs: max = 15.3s, min = 15.2s, avg = 15.2s, dev = 0.0s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 16.7s Stats over 3 runs: max = 16.7s, min = 15.0s, avg = 15.8s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 319.3s Stats over 3 runs: max = 319.3s, min = 15.8s, avg = 117.1s, dev = 143.0s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 19.4s Stats over 3 runs: max = 19.4s, min = 15.6s, avg = 16.8s, dev = 1.8s //tensorflow/compiler/xla/tests:dynamic_ops_test_cpu PASSED in 10.1s Stats over 4 runs: max = 10.1s, min = 9.3s, avg = 9.8s, dev = 0.3s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 1.2s Stats over 4 runs: max = 1.2s, min = 0.6s, avg = 0.8s, dev = 0.2s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 34.7s Stats over 4 runs: max = 34.7s, min = 18.3s, avg = 26.4s, dev = 6.2s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 57.5s Stats over 4 runs: max = 57.5s, min = 40.4s, avg = 45.7s, dev = 6.9s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 63.8s Stats over 4 runs: max = 63.8s, min = 38.9s, avg = 50.5s, dev = 10.8s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 46.1s Stats over 4 runs: max = 46.1s, min = 31.1s, avg = 36.9s, dev = 5.7s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 143.6s Stats over 4 runs: max = 143.6s, min = 131.9s, avg = 135.2s, dev = 4.9s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 12.9s Stats over 4 runs: max = 12.9s, min = 10.5s, avg = 11.8s, dev = 1.0s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 58.4s Stats over 4 runs: max = 58.4s, min = 29.1s, avg = 45.1s, dev = 13.2s //tensorflow/python/data/kernel_tests:batch_test PASSED in 32.1s Stats over 4 runs: max = 32.1s, min = 27.2s, avg = 28.9s, dev = 1.9s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 21.9s Stats over 4 runs: max = 21.9s, min = 11.7s, avg = 16.8s, dev = 4.9s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 32.2s Stats over 4 runs: max = 32.2s, min = 21.4s, avg = 26.9s, dev = 4.1s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 23.2s Stats over 4 runs: max = 23.2s, min = 10.0s, avg = 15.8s, dev = 5.9s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 24.8s Stats over 4 runs: max = 24.8s, min = 21.4s, avg = 23.3s, dev = 1.3s //tensorflow/python/data/kernel_tests:skip_test PASSED in 43.1s Stats over 4 runs: max = 43.1s, min = 28.0s, avg = 36.0s, dev = 6.3s //tensorflow/python/data/kernel_tests:take_test PASSED in 21.9s Stats over 4 runs: max = 21.9s, min = 21.0s, avg = 21.5s, dev = 0.4s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 25.5s Stats over 4 runs: max = 25.5s, min = 23.9s, avg = 24.6s, dev = 0.7s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 36.0s Stats over 4 runs: max = 36.0s, min = 26.0s, avg = 31.0s, dev = 4.9s //tensorflow/python/data/kernel_tests:zip_test PASSED in 37.5s Stats over 4 runs: max = 37.5s, min = 36.0s, avg = 36.6s, dev = 0.6s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 32.5s Stats over 4 runs: max = 32.5s, min = 22.8s, avg = 26.3s, dev = 3.7s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 52.5s Stats over 4 runs: max = 52.5s, min = 38.4s, avg = 45.0s, dev = 5.1s //tensorflow/python/framework:convert_to_constants_test PASSED in 44.3s Stats over 4 runs: max = 44.3s, min = 27.5s, avg = 34.3s, dev = 6.8s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 55.6s Stats over 4 runs: max = 55.6s, min = 52.2s, avg = 53.6s, dev = 1.3s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 16.4s Stats over 4 runs: max = 16.4s, min = 14.3s, avg = 15.1s, dev = 0.8s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 104.5s Stats over 4 runs: max = 104.5s, min = 36.4s, avg = 65.3s, dev = 28.3s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 36.7s Stats over 4 runs: max = 36.7s, min = 12.0s, avg = 21.6s, dev = 9.9s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 101.7s Stats over 4 runs: max = 101.7s, min = 21.2s, avg = 52.0s, dev = 33.0s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 34.8s Stats over 4 runs: max = 34.8s, min = 32.8s, avg = 33.9s, dev = 0.8s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 42.7s Stats over 4 runs: max = 42.7s, min = 30.9s, avg = 36.4s, dev = 4.4s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 117.1s Stats over 4 runs: max = 117.1s, min = 23.3s, avg = 64.4s, dev = 40.8s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 27.6s Stats over 4 runs: max = 27.6s, min = 26.2s, avg = 26.9s, dev = 0.5s //tensorflow/python/ops:nn_batchnorm_test_cpu PASSED in 25.0s Stats over 4 runs: max = 25.0s, min = 19.3s, avg = 21.2s, dev = 2.3s //tensorflow/python/ops:nn_fused_batchnorm_d9m_test_cpu PASSED in 21.8s Stats over 4 runs: max = 21.8s, min = 19.7s, avg = 21.1s, dev = 0.8s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 78.7s Stats over 4 runs: max = 78.7s, min = 23.3s, avg = 49.1s, dev = 19.7s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 58.9s Stats over 4 runs: max = 58.9s, min = 53.3s, avg = 56.7s, dev = 2.1s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 12.1s Stats over 5 runs: max = 12.1s, min = 11.6s, avg = 11.8s, dev = 0.2s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 16.0s Stats over 5 runs: max = 16.0s, min = 10.9s, avg = 13.1s, dev = 2.3s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 17.9s Stats over 5 runs: max = 17.9s, min = 11.7s, avg = 14.2s, dev = 2.6s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 28.8s Stats over 5 runs: max = 28.8s, min = 27.2s, avg = 27.9s, dev = 0.6s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 32.8s Stats over 5 runs: max = 32.8s, min = 11.0s, avg = 20.1s, dev = 10.1s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 11.2s Stats over 5 runs: max = 11.2s, min = 9.3s, avg = 10.2s, dev = 0.6s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 12.2s Stats over 5 runs: max = 12.2s, min = 10.6s, avg = 11.5s, dev = 0.5s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 12.0s Stats over 5 runs: max = 12.0s, min = 10.0s, avg = 10.5s, dev = 0.8s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 12.2s Stats over 5 runs: max = 12.2s, min = 11.2s, avg = 11.8s, dev = 0.4s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 24.6s Stats over 5 runs: max = 24.6s, min = 21.6s, avg = 23.2s, dev = 1.1s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 11.8s Stats over 5 runs: max = 11.8s, min = 9.9s, avg = 10.4s, dev = 0.7s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 11.3s Stats over 5 runs: max = 11.3s, min = 9.9s, avg = 10.3s, dev = 0.5s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 101.3s Stats over 5 runs: max = 101.3s, min = 18.2s, avg = 47.7s, dev = 28.6s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 116.7s Stats over 5 runs: max = 116.7s, min = 19.3s, avg = 50.8s, dev = 34.6s //tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu PASSED in 26.2s Stats over 5 runs: max = 26.2s, min = 12.3s, avg = 20.7s, dev = 6.3s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 2.7s Stats over 5 runs: max = 2.7s, min = 1.7s, avg = 2.3s, dev = 0.4s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 20.8s Stats over 5 runs: max = 20.8s, min = 18.8s, avg = 19.8s, dev = 0.6s //tensorflow/dtensor/python/tests:multi_mesh_test_cpu PASSED in 13.5s Stats over 5 runs: max = 13.5s, min = 12.1s, avg = 12.9s, dev = 0.5s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 14.2s Stats over 5 runs: max = 14.2s, min = 11.1s, avg = 13.1s, dev = 1.3s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 14.5s Stats over 5 runs: max = 14.5s, min = 13.6s, avg = 14.0s, dev = 0.3s //tensorflow/python/distribute:vars_test_2gpu PASSED in 17.7s Stats over 5 runs: max = 17.7s, min = 16.7s, avg = 17.3s, dev = 0.4s //tensorflow/python/distribute:vars_test_cpu PASSED in 23.0s Stats over 5 runs: max = 23.0s, min = 18.8s, avg = 21.5s, dev = 1.4s //tensorflow/python/eager:device_placement_test_cpu PASSED in 11.8s Stats over 5 runs: max = 11.8s, min = 10.2s, avg = 10.9s, dev = 0.5s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 132.8s Stats over 5 runs: max = 132.8s, min = 17.4s, avg = 62.2s, dev = 39.4s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 23.4s Stats over 5 runs: max = 23.4s, min = 15.7s, avg = 19.1s, dev = 2.7s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 56.9s Stats over 5 runs: max = 56.9s, min = 37.3s, avg = 47.4s, dev = 7.1s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 62.3s Stats over 5 runs: max = 62.3s, min = 58.5s, avg = 60.8s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 54.8s Stats over 5 runs: max = 54.8s, min = 53.1s, avg = 54.0s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 45.7s Stats over 5 runs: max = 45.7s, min = 39.2s, avg = 42.4s, dev = 2.5s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 43.0s Stats over 5 runs: max = 43.0s, min = 28.7s, avg = 38.1s, dev = 5.5s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 33.5s Stats over 5 runs: max = 33.5s, min = 30.8s, avg = 32.4s, dev = 1.2s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 52.4s Stats over 5 runs: max = 52.4s, min = 51.0s, avg = 51.8s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 63.7s Stats over 5 runs: max = 63.7s, min = 43.0s, avg = 56.9s, dev = 8.0s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 25.4s Stats over 5 runs: max = 25.4s, min = 22.7s, avg = 24.0s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 24.9s Stats over 5 runs: max = 24.9s, min = 22.6s, avg = 23.4s, dev = 0.8s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 133.8s Stats over 5 runs: max = 133.8s, min = 113.3s, avg = 125.0s, dev = 9.6s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 10.6s Stats over 5 runs: max = 10.6s, min = 8.2s, avg = 9.9s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 22.6s Stats over 5 runs: max = 22.6s, min = 21.5s, avg = 22.0s, dev = 0.4s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 21.0s Stats over 5 runs: max = 21.0s, min = 15.3s, avg = 17.0s, dev = 2.1s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 22.7s Stats over 5 runs: max = 22.7s, min = 12.9s, avg = 15.6s, dev = 3.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 42.6s Stats over 5 runs: max = 42.6s, min = 14.8s, avg = 21.5s, dev = 10.7s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 67.1s Stats over 5 runs: max = 67.1s, min = 27.4s, avg = 44.0s, dev = 14.2s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 17.5s Stats over 6 runs: max = 17.5s, min = 13.5s, avg = 15.6s, dev = 1.2s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 21.2s Stats over 6 runs: max = 21.2s, min = 15.4s, avg = 18.7s, dev = 1.8s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 32.4s Stats over 6 runs: max = 32.4s, min = 10.1s, avg = 20.3s, dev = 9.9s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 106.9s Stats over 6 runs: max = 106.9s, min = 11.1s, avg = 30.1s, dev = 34.4s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 56.3s Stats over 6 runs: max = 56.3s, min = 30.0s, avg = 44.1s, dev = 8.4s //tensorflow/python/ops:accumulate_n_benchmark_cpu PASSED in 9.9s Stats over 6 runs: max = 9.9s, min = 9.3s, avg = 9.6s, dev = 0.2s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 13.4s Stats over 7 runs: max = 13.4s, min = 7.6s, avg = 10.5s, dev = 2.1s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 64.3s Stats over 8 runs: max = 64.3s, min = 10.1s, avg = 28.9s, dev = 18.6s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 67.9s Stats over 8 runs: max = 67.9s, min = 10.0s, avg = 28.0s, dev = 19.3s //tensorflow/dtensor/python/tests:input_util_test PASSED in 28.6s Stats over 8 runs: max = 28.6s, min = 20.7s, avg = 25.0s, dev = 2.5s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 30.8s Stats over 8 runs: max = 30.8s, min = 10.3s, avg = 16.6s, dev = 6.9s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 30.3s Stats over 8 runs: max = 30.3s, min = 15.4s, avg = 23.6s, dev = 5.0s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 55.0s Stats over 8 runs: max = 55.0s, min = 9.3s, avg = 27.4s, dev = 16.2s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 35.7s Stats over 8 runs: max = 35.7s, min = 10.3s, avg = 16.9s, dev = 8.7s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 26.7s Stats over 8 runs: max = 26.7s, min = 9.3s, avg = 15.8s, dev = 6.1s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 24.2s Stats over 8 runs: max = 24.2s, min = 6.1s, avg = 12.2s, dev = 5.5s //tensorflow/python/data/kernel_tests:filter_test PASSED in 17.2s Stats over 8 runs: max = 17.2s, min = 14.8s, avg = 16.0s, dev = 0.8s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 46.1s Stats over 8 runs: max = 46.1s, min = 16.1s, avg = 35.4s, dev = 11.4s //tensorflow/python/data/kernel_tests:shard_test PASSED in 36.5s Stats over 8 runs: max = 36.5s, min = 24.5s, avg = 30.1s, dev = 3.9s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 68.1s Stats over 8 runs: max = 68.1s, min = 35.1s, avg = 40.3s, dev = 10.6s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 30.7s Stats over 8 runs: max = 30.7s, min = 17.8s, avg = 23.9s, dev = 3.5s //tensorflow/python/distribute/failure_handling:failure_handler_test PASSED in 72.7s Stats over 8 runs: max = 72.7s, min = 28.3s, avg = 49.2s, dev = 14.7s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 62.3s Stats over 8 runs: max = 62.3s, min = 39.9s, avg = 52.8s, dev = 7.3s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 83.5s Stats over 8 runs: max = 83.5s, min = 62.0s, avg = 73.3s, dev = 7.4s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 77.0s Stats over 8 runs: max = 77.0s, min = 50.7s, avg = 64.0s, dev = 8.6s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 60.0s Stats over 8 runs: max = 60.0s, min = 6.2s, avg = 17.5s, dev = 16.9s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 9.9s Stats over 8 runs: max = 9.9s, min = 9.2s, avg = 9.6s, dev = 0.2s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 50.6s Stats over 8 runs: max = 50.6s, min = 32.6s, avg = 39.3s, dev = 5.9s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 27.9s Stats over 8 runs: max = 27.9s, min = 15.6s, avg = 19.8s, dev = 3.5s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test FLAKY, failed in 1 out of 9 in 395.3s Stats over 9 runs: max = 395.3s, min = 14.4s, avg = 75.1s, dev = 115.8s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/gce_failure_handler_test/shard_7_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 9.4s Stats over 10 runs: max = 9.4s, min = 8.6s, avg = 8.9s, dev = 0.3s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 10.4s Stats over 10 runs: max = 10.4s, min = 9.7s, avg = 10.1s, dev = 0.3s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 13.1s Stats over 10 runs: max = 13.1s, min = 8.0s, avg = 11.1s, dev = 1.4s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 16.1s Stats over 10 runs: max = 16.1s, min = 10.4s, avg = 12.9s, dev = 1.8s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 15.6s Stats over 10 runs: max = 15.6s, min = 9.3s, avg = 12.4s, dev = 2.0s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 88.4s Stats over 10 runs: max = 88.4s, min = 43.7s, avg = 61.0s, dev = 12.8s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 78.2s Stats over 10 runs: max = 78.2s, min = 46.8s, avg = 63.3s, dev = 10.9s //tensorflow/compiler/xla/client/lib:svd_test_cpu PASSED in 35.7s Stats over 10 runs: max = 35.7s, min = 6.8s, avg = 15.1s, dev = 9.9s //tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu PASSED in 9.0s Stats over 10 runs: max = 9.0s, min = 6.7s, avg = 7.6s, dev = 0.8s //tensorflow/compiler/xla/service/cpu:cpu_runtime_test PASSED in 23.4s Stats over 10 runs: max = 23.4s, min = 0.6s, avg = 18.4s, dev = 8.9s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 19.3s Stats over 10 runs: max = 19.3s, min = 9.0s, avg = 12.8s, dev = 3.2s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 20.7s Stats over 10 runs: max = 20.7s, min = 9.1s, avg = 14.5s, dev = 3.8s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 22.7s Stats over 10 runs: max = 22.7s, min = 6.7s, avg = 13.5s, dev = 4.8s //tensorflow/python/framework:config_vgpu_test_2gpu PASSED in 15.4s Stats over 10 runs: max = 15.4s, min = 12.1s, avg = 13.6s, dev = 1.2s //tensorflow/python/framework:config_vgpu_test_cpu PASSED in 14.2s Stats over 10 runs: max = 14.2s, min = 13.7s, avg = 14.0s, dev = 0.1s //tensorflow/python/framework:function_test_cpu PASSED in 68.1s Stats over 10 runs: max = 68.1s, min = 12.6s, avg = 20.1s, dev = 16.3s //tensorflow/python/grappler:cluster_test_cpu PASSED in 10.9s Stats over 10 runs: max = 10.9s, min = 6.6s, avg = 9.7s, dev = 1.6s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 18.7s Stats over 10 runs: max = 18.7s, min = 13.1s, avg = 16.0s, dev = 1.9s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 10.6s Stats over 10 runs: max = 10.6s, min = 6.1s, avg = 8.7s, dev = 1.5s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 13.2s Stats over 10 runs: max = 13.2s, min = 8.8s, avg = 10.6s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 165.0s Stats over 10 runs: max = 165.0s, min = 98.2s, avg = 141.4s, dev = 18.5s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 130.4s Stats over 10 runs: max = 130.4s, min = 4.9s, avg = 20.6s, dev = 36.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 35.9s Stats over 10 runs: max = 35.9s, min = 13.0s, avg = 23.7s, dev = 7.4s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 28.9s Stats over 10 runs: max = 28.9s, min = 10.7s, avg = 19.2s, dev = 7.6s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 45.2s Stats over 10 runs: max = 45.2s, min = 10.6s, avg = 18.1s, dev = 12.9s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 18.3s Stats over 10 runs: max = 18.3s, min = 16.0s, avg = 17.1s, dev = 0.7s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 31.9s Stats over 10 runs: max = 31.9s, min = 10.4s, avg = 15.3s, dev = 8.1s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 108.1s Stats over 10 runs: max = 108.1s, min = 21.6s, avg = 63.6s, dev = 40.9s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 61.3s Stats over 10 runs: max = 61.3s, min = 12.7s, avg = 19.8s, dev = 14.2s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 17.4s Stats over 10 runs: max = 17.4s, min = 12.5s, avg = 14.7s, dev = 1.6s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 116.9s Stats over 10 runs: max = 116.9s, min = 96.7s, avg = 107.8s, dev = 7.5s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 24.6s Stats over 10 runs: max = 24.6s, min = 19.1s, avg = 21.7s, dev = 1.8s //tensorflow/python/saved_model:load_test_cpu PASSED in 75.8s Stats over 10 runs: max = 75.8s, min = 39.9s, avg = 46.4s, dev = 10.2s //tensorflow/compiler/tests:fft_test_cpu PASSED in 28.4s Stats over 12 runs: max = 28.4s, min = 13.1s, avg = 19.7s, dev = 5.9s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 4.4s Stats over 12 runs: max = 4.4s, min = 2.6s, avg = 3.5s, dev = 0.6s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 23.7s Stats over 12 runs: max = 23.7s, min = 9.4s, avg = 14.0s, dev = 4.5s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 14.8s Stats over 12 runs: max = 14.8s, min = 9.2s, avg = 10.8s, dev = 1.6s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 15.6s Stats over 12 runs: max = 15.6s, min = 10.6s, avg = 13.7s, dev = 1.6s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 21.3s Stats over 12 runs: max = 21.3s, min = 17.6s, avg = 19.7s, dev = 1.0s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 20.1s Stats over 12 runs: max = 20.1s, min = 11.1s, avg = 15.5s, dev = 3.1s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 228.8s Stats over 12 runs: max = 228.8s, min = 17.4s, avg = 54.1s, dev = 58.6s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 228.5s Stats over 12 runs: max = 228.5s, min = 16.3s, avg = 53.9s, dev = 58.5s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 47.6s Stats over 15 runs: max = 47.6s, min = 13.8s, avg = 31.0s, dev = 8.6s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 54.6s Stats over 15 runs: max = 54.6s, min = 14.1s, avg = 20.0s, dev = 10.4s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 12.4s Stats over 16 runs: max = 12.4s, min = 6.2s, avg = 8.9s, dev = 2.0s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 14.8s Stats over 16 runs: max = 14.8s, min = 8.3s, avg = 11.1s, dev = 1.6s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 15.1s Stats over 16 runs: max = 15.1s, min = 7.1s, avg = 10.8s, dev = 2.3s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 14.9s Stats over 16 runs: max = 14.9s, min = 4.4s, avg = 9.7s, dev = 2.8s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 28.4s Stats over 16 runs: max = 28.4s, min = 11.0s, avg = 19.6s, dev = 4.4s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 34.1s Stats over 16 runs: max = 34.1s, min = 10.9s, avg = 14.8s, dev = 5.4s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 12.0s Stats over 16 runs: max = 12.0s, min = 7.6s, avg = 9.0s, dev = 1.3s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 15.4s Stats over 16 runs: max = 15.4s, min = 10.3s, avg = 13.2s, dev = 1.8s //tensorflow/python/ops:image_ops_test_cpu PASSED in 25.2s Stats over 16 runs: max = 25.2s, min = 13.8s, avg = 17.3s, dev = 3.1s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 81.7s Stats over 17 runs: max = 81.7s, min = 6.0s, avg = 33.9s, dev = 26.4s //tensorflow/python/data/kernel_tests:map_test PASSED in 64.8s Stats over 19 runs: max = 64.8s, min = 34.0s, avg = 44.7s, dev = 7.0s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 11.0s Stats over 20 runs: max = 11.0s, min = 8.4s, avg = 9.3s, dev = 0.7s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 10.1s Stats over 20 runs: max = 10.1s, min = 8.1s, avg = 9.1s, dev = 0.6s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 33.8s Stats over 20 runs: max = 33.8s, min = 28.0s, avg = 29.8s, dev = 1.5s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 15.3s Stats over 20 runs: max = 15.3s, min = 5.7s, avg = 9.4s, dev = 2.0s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 19.7s Stats over 20 runs: max = 19.7s, min = 12.6s, avg = 16.0s, dev = 1.9s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 8.0s Stats over 20 runs: max = 8.0s, min = 6.6s, avg = 7.2s, dev = 0.4s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 16.8s Stats over 20 runs: max = 16.8s, min = 11.5s, avg = 13.6s, dev = 1.3s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 12.1s Stats over 20 runs: max = 12.1s, min = 9.8s, avg = 10.9s, dev = 0.5s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 14.6s Stats over 20 runs: max = 14.6s, min = 6.5s, avg = 11.1s, dev = 2.2s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 39.8s Stats over 20 runs: max = 39.8s, min = 8.1s, avg = 16.4s, dev = 10.1s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 43.9s Stats over 20 runs: max = 43.9s, min = 16.9s, avg = 31.4s, dev = 9.4s //tensorflow/python/kernel_tests:metrics_test PASSED in 63.0s Stats over 20 runs: max = 63.0s, min = 10.8s, avg = 29.1s, dev = 16.3s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 12.4s Stats over 20 runs: max = 12.4s, min = 7.4s, avg = 9.7s, dev = 1.6s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 17.0s Stats over 20 runs: max = 17.0s, min = 9.1s, avg = 11.4s, dev = 2.2s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 53.9s Stats over 20 runs: max = 53.9s, min = 4.6s, avg = 19.3s, dev = 15.2s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 110.9s Stats over 20 runs: max = 110.9s, min = 29.3s, avg = 58.1s, dev = 22.6s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 11.6s Stats over 20 runs: max = 11.6s, min = 5.4s, avg = 7.3s, dev = 1.6s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 23.7s Stats over 20 runs: max = 23.7s, min = 10.1s, avg = 17.5s, dev = 3.6s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 123.9s Stats over 20 runs: max = 123.9s, min = 37.2s, avg = 89.3s, dev = 29.9s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 24.2s Stats over 20 runs: max = 24.2s, min = 4.4s, avg = 11.5s, dev = 6.6s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 26.6s Stats over 20 runs: max = 26.6s, min = 8.7s, avg = 16.4s, dev = 6.5s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 19.5s Stats over 20 runs: max = 19.5s, min = 15.2s, avg = 17.5s, dev = 1.2s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 69.0s Stats over 20 runs: max = 69.0s, min = 10.4s, avg = 29.9s, dev = 19.7s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 23.7s Stats over 20 runs: max = 23.7s, min = 13.5s, avg = 15.8s, dev = 2.1s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 34.8s Stats over 24 runs: max = 34.8s, min = 13.6s, avg = 21.9s, dev = 6.2s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 23.9s Stats over 24 runs: max = 23.9s, min = 4.7s, avg = 12.9s, dev = 5.2s //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 10.4s Stats over 25 runs: max = 10.4s, min = 8.4s, avg = 9.4s, dev = 0.5s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 38.7s Stats over 25 runs: max = 38.7s, min = 7.5s, avg = 13.2s, dev = 8.3s //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 10.0s Stats over 30 runs: max = 10.0s, min = 7.6s, avg = 8.6s, dev = 0.7s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 30.0s Stats over 30 runs: max = 30.0s, min = 12.4s, avg = 16.5s, dev = 6.6s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 1732.0s Stats over 30 runs: max = 1732.0s, min = 7.5s, avg = 65.7s, dev = 309.4s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 10.3s Stats over 30 runs: max = 10.3s, min = 6.4s, avg = 7.9s, dev = 1.0s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 37.2s Stats over 30 runs: max = 37.2s, min = 26.3s, avg = 31.2s, dev = 2.6s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 28.9s Stats over 31 runs: max = 28.9s, min = 9.5s, avg = 14.5s, dev = 6.2s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 9.3s Stats over 32 runs: max = 9.3s, min = 6.4s, avg = 7.6s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 27.3s Stats over 32 runs: max = 27.3s, min = 4.8s, avg = 13.4s, dev = 6.3s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 17.5s Stats over 32 runs: max = 17.5s, min = 4.4s, avg = 11.9s, dev = 3.5s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 65.8s Stats over 32 runs: max = 65.8s, min = 39.6s, avg = 51.6s, dev = 5.9s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 9.3s Stats over 40 runs: max = 9.3s, min = 7.3s, avg = 8.4s, dev = 0.5s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 11.6s Stats over 40 runs: max = 11.6s, min = 7.9s, avg = 9.7s, dev = 0.7s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 14.2s Stats over 40 runs: max = 14.2s, min = 7.5s, avg = 9.9s, dev = 1.3s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 12.2s Stats over 40 runs: max = 12.2s, min = 6.7s, avg = 9.0s, dev = 1.4s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.9s Stats over 48 runs: max = 1.9s, min = 0.4s, avg = 0.6s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 59.2s Stats over 50 runs: max = 59.2s, min = 24.4s, avg = 46.4s, dev = 8.6s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 20.3s Stats over 50 runs: max = 20.3s, min = 3.7s, avg = 10.9s, dev = 4.4s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 18.0s Stats over 50 runs: max = 18.0s, min = 4.0s, avg = 10.2s, dev = 3.6s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 49.9s Stats over 50 runs: max = 49.9s, min = 4.0s, avg = 11.4s, dev = 12.0s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 24.5s Stats over 50 runs: max = 24.5s, min = 3.8s, avg = 7.8s, dev = 4.4s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 12.4s Stats over 50 runs: max = 12.4s, min = 9.1s, avg = 10.7s, dev = 0.8s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 26.5s Stats over 50 runs: max = 26.5s, min = 11.5s, avg = 16.9s, dev = 5.1s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 19.3s Stats over 50 runs: max = 19.3s, min = 9.2s, avg = 13.3s, dev = 2.4s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 15.2s Stats over 50 runs: max = 15.2s, min = 5.4s, avg = 9.0s, dev = 2.4s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 11.7s Stats over 50 runs: max = 11.7s, min = 4.2s, avg = 6.6s, dev = 1.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 11.4s Stats over 50 runs: max = 11.4s, min = 4.3s, avg = 9.4s, dev = 2.1s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 30.2s Stats over 50 runs: max = 30.2s, min = 7.7s, avg = 14.9s, dev = 5.5s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 15.1s Stats over 50 runs: max = 15.1s, min = 4.2s, avg = 8.6s, dev = 2.9s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 32.6s Stats over 50 runs: max = 32.6s, min = 4.0s, avg = 7.7s, dev = 4.7s Executed 3928 out of 3928 tests: 3928 tests pass. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.